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This open access book provides an introduction to Generative Artificial 
Intelligence and four cutting-edge technologies that it enables — humanoid 
robots, deepfakes, augmented reality, and virtual reality. 

Experts predict that in five years’ time, more than 90 per cent of all 
digital content will be wholly or partially AI generated. In a synthetic 
society, it may no longer be possible to establish what is real and what 
is not. Although they are only in their relative infancy, these technolo- 
gies can already produce content that is indistinguishable from authentic 
material. The impact of this new reality on democracy, the judicial 
system, the functioning of the press, as well as on personal relationships, 
might be unprecedented. 

The author describes the inner workings of each of these technolo- 
gies and maps their positive uses in the fields of education, health and 
entertainment; conceptualises their negative uses for fraud, deception, 
exploitation and identity-theft; and explores their deeper effects on the 
post-truth society, the privatisation of the public sphere, and the loss of 
individual autonomy and societal trust. 

The book evaluates how the current European legal paradigm applies 
to these technologies, focussing on the right to privacy and data protec- 
tion, intellectual property, freedom of expression, procedural law, tort 
law, consumer and competition law, and the regulation of AI. It discusses 
regulatory alternatives to solve existing regulatory gaps and shows that 
there are no easy answers. 
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1 
A Map to This Book 


We all know that Artificial Intelligence can do amazing things. And I think we 
do not talk enough about what Artificial Intelligence is able to do to improve 
our daily lives. For example, if we look at the health care sector, we know 
that we use already now Artificial Intelligence for, for example, better diagno- 
ses and earlier diagnoses. And better and earlier diagnoses are crucial when 
you treat for example cancer — or we use robots for precision surgery. But 
Artificial Intelligence is also key for us when we want to reach our goal to be 
climate neutral in 2050. Just think of precision farming: Precision farming, 
Al-driven, enables us to reduce pesticides, enables us to reduce fertilisers, for 
example. Or if you think of smart heating: Smart heating, Al-driven again, 
is saving millions and millions of tonnes of oil for example and therefore 
reducing the CO2 footprint ... We want citizens to trust the new technology. 
And technology is always neutral, it depends on what we make with it. And 
therefore, we want the application of these new technologies to deserve the 
trust of our citizens. This is why we are promoting a responsible, human- 
centric approach to Artificial Intelligence.' 

Al is a general technology that is accessible, powerful and adaptable for a vast 
range of uses — both civilian and military. And it is moving faster than even 
its developers anticipated. So we have a narrowing window of opportunity 
to guide this technology responsibly. I believe Europe, together with partners, 
should lead the way on a new global framework for Al, built on three pillars: 
guardrails, governance and guiding innovation? 


Ursula von der Leyen, European Commission, European Union 


1.1. INTRODUCTION 


S STRESSED BY Ursula von der Leyen, the President of the 
European Commission, Artificial Intelligence (AI) may radically 
change our lives and society as we know it. There will be oppor- 
tunities and chances, but also challenges, hurdles and dangers. This book 
will focus on Generative AI, or APs capacity to generate content, speech 
and action that is indistinguishable from human output. This chapter 
illustrates the myriad of questions and dilemmas that are triggered by 
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synthetic reality through one simple example of Al-generated content 
(section 1.2) and continues by introducing Generative AI and the four 
synthetic technologies central to this book: Humanoid Robots (HRs), 
Deepfakes (DFs), Augmented Reality (AR) and Virtual Reality (VR) 
(section 1.3). A description follows of what this book has to offer 
(section 1.4) as well as where its limitations lie (section 1.5). Finally, an 
overview of this book’s content is provided (section 1.6). 


1.2. WELCOME TO THE SYNTHETIC SOCIETY 


Imagine it is 2034 and DF technology has developed as predicted. DFs 
can be produced in under a minute by any citizen anywhere in the world, 
provided they have access to a smartphone or a computer. DFs have 
become so realistic that humans cannot distinguish them from authentic 
material. Even algorithmic detection programmes can only do so in about 
50 per cent of cases. Suppose someone posts a sensational video on a 
X-like platform, showing several white men desecrating the Quran. In no 
time, the clip generates attention. The left-wing media is outraged, anon- 
ymous users post racist comments and the right-wing media publishes an 
interview with a critic of Islam who claims that the holy book is danger- 
ous and sanctions violence. Comedy shows make references to it. But is 
the video real? As a citizen, by 2034, you will have understood how easy 
it is to generate DFs and may have fooled people yourself. There will 
have been instances in the past where a media frenzy has turned out to 
be unjustified because the video, audio or text was later discovered to be 
fake. What do you do as a citizen? Do you ignore the news altogether, do 
you try to make your own assessment of its authenticity, or do you only 
trust reports that have appeared through traditional media channels that 
invest in verification procedures? 

The rise of hybrid and synthetic media puts additional pressure on the 
mainstream media. Trust in these institutions is already waning: people 
are increasingly selecting sources that confirm their established world- 
views, and more and more people are suffering from ‘reality fatigue’. 
If the media are offered content several times a week that is found to 
be inaccurate, they may be forced to completely abandon working with 
User-Generated Content, due to the cost and the time that this analysis 
requires. This would mean that valuable content by citizen journalists 
would be ignored. In addition, traditional media may find that sensa- 
tional content will be posted on platforms with lower quality standards, 
meaning traditional outlets are always two steps behind, and they may 
lose out on views and revenue. 
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The rise of DFs could also have a major effect on the courtroom. 
DFs are already being used as evidence in court cases. This is a trend 
that is likely to have gained momentum by 2034. Mistakes are likely to 
be made, which may require courts to verify all content for authentic- 
ity. This may trigger a range of questions. Should the prosecution prove 
that the evidence is authentic or is it up to the defence to show that it is 
not? What standards should be used for authenticity? And what if an AI 
detection program suggests that the likelihood that material is authentic 
is 68 per cent? Not only will the verification process affect the duration 
of court cases, but there will also always be ‘plausible deniability’. A 
person will always be able to maintain that their conviction was based 
on fabricated evidence. 

As a final example, the vast majority of DFs are sexual in nature: DF 
porn. Clips often depict celebrities, but they also show young girls still in 
high school performing sexual activities they did not carry out. A high 
number of teenage girls are already victims of online slut-shaming or 
other forms of online sexual harassment. These practices are sometimes 
referred to as ‘slut-shaming 2.0’, both because of the scale and intensity 
of online images and because it is often difficult to remove such content 
or escape from it. Through DF technology, these practices could be taken 
to the next level, or what may be called ‘slut-shaming 3.0’. Experts fear 
the effect this might have on young girls’ development and the societal 
position of women. 

These are only some of the broader questions and themes raised by 
one of the technologies central to this book. As will become apparent, 
the other technologies discussed here raise similar complex questions: do 
we want to live in a society in which the primary social contact point 
of an increasing number of elderly people is a humanoid robot? Do we 
want to gamify warfare through AR? What happens if children who feel 
socially isolated prefer to live in VR instead of engaging with their peers 
in Physical Reality (PR)? These and other questions are addressed in this 
book. In particular, this volume assesses what role the legal regime can 
play in safeguarding important values that act as the pillars of our society, 
such as shared truth, individual autonomy and the democratic rule of law. 


1.3. SETTING THE STAGE 


The world is increasingly mediated through and affected by AI. Of all 
contemporary technological advances, Generative AI stands out. APs 
capability to generate audio, video, text and other content will have a 
significant effect on reality and how we perceive it. Various trends, such 
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as developments in AI Robotics, DFs, AR and VR, the four synthetic 
technologies that will be central to this book, have meant that an ever- 
larger number of our everyday experiences are Al-generated: our reality 
is becoming ever more synthetic. 

Humanoid Robots: HRs have the physical appearance of a human. 
They have the motoric skills to walk and talk like humans. They run on 
advanced software, which grants them human-like cognitive functions. 
To this end, HRs are equipped with sensors to gather the same input as 
humans. They are connected to the internet and have in-built cutting- 
edge technology, such as facial, object and emotion recognition. HRs 
are self-learning and act autonomously, using AI and Machine Learning. 
Large Language Models (LLMs) not only enhance their verbal capaci- 
ties, but also allow them to better respond to questions or tasks.* LLMs 
can also power chat programs, for example, replacing customer service 
employees. Social interaction with HRs is becoming more fluid, natural 
and intuitive. Even back in 1996, when the Tamagotchi was introduced 
(a small, egg-shaped computer that ran a program with a pet that the 
owner took care of),* it was clear how quickly people became attached 
to inanimate objects. Now, with advances in design, functionality and 
sociability, people are increasingly interacting with and becoming fond of 
their robots. This is not limited to children or adolescents who prefer the 
comfort of AI friends over their peers, or men who find their life partners 
in HRs; it extends to elderly people who use AI as conversational part- 
ners, always available for social interaction, and more.’ Experts predict 
that it will be increasingly difficult to distinguish robots from humans as 
technology and design progress, and the rise of Large Language Models 
has meant that communicating with HRs and chatbots can already be an 
exciting experience. 

Deepfakes: DFs are content (video, audio or otherwise) that is wholly 
or partially fabricated, or existing content that has been manipulated, 
using AI. Several technologies can be used for this, but the most popular 
is based on Generative Adversarial Networks (GAN). As with HRs, the 
ultimate goal for DFs is that Al-generated content cannot be distinguished 
from authentic material. In just a few years, DF technology has advanced 
rapidly in terms of quality, speed of production and cost-efficiency, so 
much so that humans are no longer able to distinguish fake from authentic 
content. A fake video can be generated in a minute or two. Although the 
technology is still in its relative infancy, there are signs that people already 
find DF faces of non-existent people more trustworthy than those of real 
people.® Many DFs are used for relatively trivial purposes, but because in a 
DE real people may appear to be doing and saying things they never did or 
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said, they can have a real and significant impact. DFs are used in warfare 
(for example, generals who appear to order their troops to lay down their 
arms), politics (for example, videos of political leaders swearing and curs- 
ing at their opponents just before election day) and the courtroom (for 
example, a mother who introduces a fake video of the father acting inap- 
propriately towards their children in a custody case). 

Augmented Reality: AR is reality as perceived by the user of particu- 
lar piece of equipment that adds layers to reality using AI. Thus, one’s 
perception of reality is grounded in PR, but Al-generated sensory (audio, 
video, olfactory etc) input is received on top of that or instead of certain 
parts of it. AR is often used in the same breath as ‘gamification’, in which 
an activity or task is made ‘fun’ by adding colours, sounds and other 
stimuli to reality. However, the technology can be deployed in a wide 
variety of ways, including for professional purposes. It can be used to 
show a route while walking in the forest by putting yellow smiley faces 
on the right track; it might be used in war zones to replace the sight of 
blood with more pleasant stimuli. In Smart Cities, Al is used to adjust the 
smell in the streets. A tangerine scent can be sprayed in areas if predictive 
programming has found a high likelihood of nightlife aggression. The 
smell of fruit is known to have a calming effect on people.’ 

Virtual Reality: In its ideal form, when a user is in VR, all sensory 
input is Al-generated. Most current forms of VR focus on creating 
Al-generated input for just one of our senses, namely sight. However, 
there are also experiments with sound, smell and touch. Although a user 
enters the new reality consciously, its virtuality is generally perceived as 
almost real, which is different from most forms of AR, where users are 
generally aware of the artificiality of the added sensory layers. VR can 
have real and concrete effects on people when they return to Physical 
Reality, such as when being treated for anxiety disorders or chronic 
pain. VR is also used for gaming, for treatment of people suffering from 
dissociative disorders, for military training and for educational purposes — 
for example, allowing children to participate in democratic debates in 
ancient Athens. 


1.4. WHAT THIS BOOK OFFERS 


These are just a few trends that will influence our sense and perception 
of reality. Although these technologies are still in their relative infancy, 
they have already had real and tangible consequences for the personal 
lives of many and society at large. Even if these technologies advance in 
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accordance with the most conservative scenarios, reality will be increas- 
ingly mediated, affected, altered and substituted by Al-generated sensory 
input. Consequently, this book’s central theme is reality. 

Focusing on the implications of Synthetic Reality, this book discusses 
four fields of application: HRs, DFs, AR and VR. The reason for choos- 
ing these four is that they represent different technological currents that 
affect our perception of reality. An HR is placed in PR and acts and talks 
like a human. A DF is an AI generated representation of reality, though a 
distorted one. AR takes reality as a basis, but adds Al-generated stimuli 
to the user’s perception. Through VR, the user’s perception of reality is 
fully Al-generated, while her body remains in PR. 

This book sets out to describe the effects of Al-generated content 
on reality and our perception of it. It describes the various ongoing 
technological developments and discusses the main fields of applica- 
tion of HRs, DFs, AR and VR. It discusses both the potential positive 
and negative effects of these applications, and homes in on the larger 
societal questions the technologies raise. It describes the current regula- 
tory regime and evaluates how it applies to the technologies and their 
main use cases. Several regulatory gaps become apparent: points where 


Augmented 
Reality 


Figure 1.1 The four AI technologies discussed in this book 
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by AI 


Figure 1.2 The impact these technologies have on reality 


the negative consequences of the technologies cannot be adequately 
addressed through the current legal standards. The book closes by 
discussing several regulatory alternatives to the current regime that could 
solve these gaps, and discusses the merits and pitfalls of each of these 
alternatives. In the end, there may be more questions than answers. 

This book is written for a broad audience, but caters in particular 
to lawyers, legal scholars, regulators, and law and technology students. 
Hence, its focus is on the consequences of synthetic technologies for 
society and how the legal regime could or should be adapted so that 
it can adequately address the harmful effects on personal and societal 
interests. After finishing the book, the reader should have a grasp on the 
basic qualities of the four technologies and their main fields of applica- 
tion, the beneficial and harmful consequences of their use in practice, 
and the bigger societal questions they raise. The reader should also have 
an understanding of how the current regulatory regime applies to the 
various technologies, what regulatory gaps there are and what poten- 
tial regulatory alternatives could improve the prevailing legal standards. 
Above all, it should be clear that there often are no perfect solutions to 
the many uneasy questions Generative AI raises. 
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There are several limitations that come with this approach. 

First, the terms ‘synthetic reality’, ‘Generative AI’, ‘Humanoid Robot’, 
‘Deepfake’, ‘Augmented Reality’ and ‘Virtual Reality’ do not lend them- 
selves to exact definitions. When is fake material to be considered a DF? 
Does it depend on the technology used to produce the fake, whether the 
consumer believes it to be real, or its deep impact on the perception of 
reality? When exactly does reality become so augmented that it should be 
called VR? If 90 per cent of what a person sees consists of Al-generated 
layers, is that still AR? What if VR is an exact copy of PR — is that VR or 
more like an interactive film? At what point does a robot look, talk, think 
and act so much like a human that it should be considered humanoid? All 
these definitions depend on sliding scales rather than binary distinctions; 
they are best treated as ideal concepts that may never be fully realised. 

Second, following the lack of binary distinctions, the concepts 
used in this book have significant overlap and are not always distin- 
guishable in an absolute fashion. For example, the Metaverse makes use 
of AR, VR, the Internet of Things and tactile internet applications.® 
Sometimes, AR is based on a fully representational system, represent- 
ing reality through a video screen, while adding an AI layer to the 
video recording of reality. HRs use a lot of the same technology as VR. 
Apple’s Vision Pro uses VR, AR and DF technology. Japan’s Moonshot 
program, which has the aim of the ‘realization of a society in which 
human beings can be free from limitations of body, brain, space, and 
time by 2050’,? depends on multiple synthetic technologies. In this 
book, the four technologies will be used as a rough division between AI 
impersonated reality, AI (mis)represented reality, AI altered reality and 
Al replaced reality. 

Third, there are other, connected developments that do not take 
centre stage here, but are discussed where relevant. Each of the technolo- 
gies depends on enabling hardware and software. Modern AI robotics, 
for example, requires an understanding of AI, Machine Learning, Large 
Language Models and the field of robotics, but also of the Internet of 
Things, facial and emotion recognition, and speech analysis. AR and VR 
depend on a wide variety of hardware for sensory input and output. DFs 
are typically generated using Generative Adversarial Networks, which 
also have a wide variety of other uses. All these adjoining technological 
concepts are only elaborated upon to the extent necessary for under- 
standing the four central technologies. 
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Fourth, what connects the technologies in this book is that they are 
Al-driven. However, they may have effects that could be achieved through 
non-Al-mediated means. Glasses have allowed a distorted view of real- 
ity for decades. Smart Cities may be Al-driven, but spraying a tangerine 
scent is not structurally different from non-Al mediated forms of scent- 
nudging, such as perfumes sprayed in shops to arouse appetite. DFs 
may be more realistic, but photography and film have allowed for the 
manipulation of the truth since their introduction. Thus, although the 
technologies discussed in this book are certainly potentially disruptive, 
early forms of altered, manipulated and enhanced perceptions of reality 
exist in abundance. 

Fifth, by the time this book is published, many concrete examples, 
applications and descriptions of the technologies may be outdated. All 
four of the technologies central to the discussion here are advancing at 
rapid speeds. This is why the description of societal effects and the analy- 
sis of the regulatory gaps and solutions steer clear of anecdotal evidence 
and incidental problems. Rather, the focus here is on the more general 
and structural changes that these technologies will bring about and the 
types of applications they enable. New and unforeseen developments 
and applications not covered by this book may take place, although spec- 
ulation over how the technologies may evolve is included where relevant. 

Sixth, this book describes the technologies and their prospected 
development over the coming years. Since the technologies are still in 
their relative infancy, most sources about their potential harms and bene- 
fits come from companies and start-ups that develop the technologies or 
market their applications, or from tech-savvy journalists and academ- 
ics. Thus, the description of the technologies’ potential applications 
(Chapter 2) and their functionality and design (Chapter 3) may have a 
particular bias towards positive use cases and an unrealistic perception 
of their effectiveness. Still, especially with AI robots and DFs, consider- 
able academic and journalistic attention has been devoted to negative 
scenarios as well. 

Seventh, this means that although the various technologies are 
already applied in practice, and the general idea is that they will evolve 
and have an ever greater societal impact, which will be taken as the basic 
assumption of this book, this is not a given. As with other technolo- 
gies, most revolutionary predictions have not come to pass; this is no less 
true for the technologies described in this book. AR, for example, has 
been promoted ever since the 1990s, but has not been the success early 
adaptors and investors had believed it would be. AI more generally has 
not delivered on its early promises; we are already witnessing the fourth 
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generation of AI, which means that the former three have only yielded 
marginal results. This book takes a hypothetical scenario: what if the 
technologies do become relatively successful; how would they affect the 
world and how could the legal regime be adapted to adequately address 
their effects? Although the technologies will not develop along the lines 
now predicted, these predictions are the best there is to go on for now. It 
is important to think through this scenario because, in 10 years’ time, if 
the technologies are as successful as we now believe they will be, it may 
be too late to introduce regulations to curb their effects. 

Eighth, this book describes HRs, DFs, AR and VR through the lens 
of their effects on reality and our perception of it. Thus, it does not 
provide an exposé on AI robotics or a full description of VR’s poten- 
tial in the gaming industry or any other specific application. Rather, the 
descriptions are limited to aspects that have a bearing on this book’s 
central theme: reality. The cases discussed serve as examples of how 
Al-generated reality may change our perception of ourselves, others and 
the world around us. 

Ninth, this book’s focus is legal in nature. The technologies are 
described from this perspective; the aspects and applications of the 
technologies that are relevant to the legal analysis are highlighted, while 
others may be ignored or only touched upon briefly. Thus, this book 
should not be taken as providing an exhaustive description of these tech- 
nologies. It describes the technologies in a way that any well-informed 
reader can grasp their significance. The descriptions of the technologies 
and their applications (Chapters 2 and 3) summarise the most impor- 
tant developments in this field as discussed by other academics and will 
closely follow their findings. The novelty of this book lies in its societal, 
legal and regulatory analysis (Chapters 4, 5 and 6). 

Tenth, the description of the legal framework is based on European law, 
provided by the Council of Europe, especially through the European 
Convention on Human Rights and the jurisprudence of the European 
Court of Human Rights, and by the European Union, through laws like 
the General Data Protection Regulation, the Artificial Intelligence Act 
and the Digital Services Act. Not all legal domains are covered by the 
Council of Europe or the European Union, such as criminal law and tort 
law. These legal domains are largely left to the regulatory discretion of 
Member States, such as France, Germany and the Netherlands. This book 
largely leaves these domains untouched; if they play a role, the discussion 
here remains somewhat abstract, covering the basic concepts or doctrines 
that are part of most national legal regimes. The same applies to matters 
of procedural law and law enforcement. This is covered, but only from 
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a general perspective. In this way, the book avoids being relevant only to 
legal experts from one particular country. 


1.6. OVERVIEW 


This book is constructed as follows. Chapter 2 presents an overview of 
the most important fields of application of synthetic technologies, as 
well as their positive and negative consequences. After an introduction 
(section 2.1), it focuses on the use cases for HRs (section 2.2), DF tech- 
nology (section 2.3), AR (section 2.4) and VR (section 2.5), followed by 
a wrap-up (section 2.6). 

Chapter 3 explains the basic design and architecture behind the four 
technologies. After an introduction (section 3.1), it provides the reader 
with an explanation of the hard- and software used for HRs (section 3.2), 
DFs (section 3.3), AR (section 3.4) and VR (section 3.5). This is followed 
by a conclusion (section 3.6). 

Chapter 4 describes the broader societal effects of these technologies 
with regard to basic values shared throughout Western society, such as 
truth and trust (section 4.2), human recognition and social interaction 
(section 4.3), autonomy and equality (section 4.4), and the rule of law 
and protection of (non-)human rights (section 4.5). These sections are 
preceded by an introduction (section 4.1) and followed by a conclusion 
(section 4.6). 

Chapter 5 evaluates how the current regulatory framework applies to 
the four synthetic technologies and their applications, which also exposes 
the most relevant regulatory gaps. After an introduction (section 5.1), 
it focuses on data protection law (section 5.2), freedom, privacy and 
property rights (section 5.3), rules on the production and use of technol- 
ogy and the transfer of data (section 5.4), and issues revolving around 
liability, law enforcement and procedural law (section 5.5). The final part 
summarises the chapter’s main findings (section 5.6). 

As Chapter 5 raises a great many questions regarding the applica- 
bility of the current legal framework on synthetic technologies and the 
need for revisions, Chapter 6 lays the groundwork for addressing these 
challenges. This book does not provide sweeping statements or easily 
implementable solutions. Rather, its main objective is to raise regulatory 
and societal questions, showing what is at stake with the rise of these 
technologies and that every regulatory option has both its merits and 
pitfalls. There are no easy solutions, which is what makes regulating the 
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synthetic society fascinating, essential and complex. After an introduc- 
tion (section 6.1), this chapter will recount the main findings of this book 
(section 6.2) and map some of the proposals that experts have suggested 
for dealing with the moral, legal and societal questions raised by synthetic 
technology (section 6.3). It will also provide a model through which regu- 
lators can address the regulatory questions (section 6.4) and, finally, will 
provide concluding remarks (section 6.5). 


2 


Applications and Effects of 
Synthetic Technologies 


Al could help people with improved health care, safer cars and other trans- 
port systems, tailored, cheaper and longer-lasting products and services. It 
can also facilitate access to information, education and training. The need for 
distance learning became more important because of the Covid-19 pandemic. 
AI can also make workplace safer as robots can be used for dangerous parts 
of jobs, and open new job positions as Al-driven industries grow and change. 
For businesses, Al can enable the development of a new generation of prod- 
ucts and services, including in sectors where European companies already 
have strong positions: green and circular economy, machinery, farming, 
healthcare, fashion, tourism. It can boost sales, improve machine mainte- 
nance, increase production output and quality, improve customer service, 
as well as save energy. 11-37% Estimated increase of labour productivity 
related to AI by 2035. Al used in public services can reduce costs and offer 
new possibilities in public transport, education, energy and waste manage- 
ment and could also improve the sustainability of products. In this way 
AI could contribute to achieving the goals of the EU Green Deal. 1.5-4% 
Estimate of how much AI could help reduce global greenhouse emissions 
by 2030. Democracy could be made stronger by using data-based scrutiny, 
preventing disinformation and cyber attacks and ensuring access to quality 
information. Al could also support diversity and openness, for example by 
mitigating the possibility of prejudice in hiring decisions and using analyti- 
cal data instead. Al is predicted to be used more in crime prevention and 
the criminal justice system, as massive data sets could be processed faster, 
prisoner flight risks assessed more accurately, crime or even terrorist attacks 
predicted and prevented. It is already used by online platforms to detect and 
react to unlawful and inappropriate online behaviour. In military matters, Al 
could be used for defence and attack strategies in hacking and phishing or to 
target key systems in cyberwarfare.' 


European Parliament, European Union 
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2.1. INTRODUCTION 


HE EUROPEAN PARLIAMENT has mapped a high number of opportu- 

nities for Generative AI, as well as dangers, including those relating 

to privacy and security, job loss and discrimination. This chapter 
discusses the main use cases and their potential beneficial and detrimental 
effects for the four technologies that are central to this book: Humanoid 
Robots (HRs) (section 2.2), Deepfakes (DFs) (section 2.3), Augmented 
Reality (AR) (section 2.4) and Virtual Reality (VR) (section 2.5). 
The chapter closes with a conclusion (section 2.6). As such, this chapter 
lays the groundwork for Chapter 3 (describing the architecture and design 
of the four technologies), Chapter 4 (homing in on the larger societal 
effects and challenges synthetic reality triggers) and Chapter 5 (evaluating 
synthetic technologies and their use from a legal perspective). 


2.2. HUMANOID ROBOTS 


Robots are increasingly being used for industrial and manufacturing jobs, 
especially for repetitive and precision tasks.* For example, teleoperated 
robot-assisted surgery provides surgeons with additional advantages in 
minimally invasive surgery with improved precision, dexterity and visu- 
alisation.* AI robots can perform surgical tasks autonomously,’ but they 
mostly operate in human-robot teams.° Humans may also be equipped 
with robotic material, such as if a person who suffers from partial paraly- 
sis uses a robot exoskeleton to bring back normal arm functionality. Some 
applications allow humans to ‘be’ the robot, that is, to steer its activities 
through telepresence.’ Since robots are better and more efficient at certain 
types of jobs, robots substituting humans may lead to a reduction in waste 
production.® Similarly, in restaurants, robots can be used for hygienic 
reasons.’ They can help with lifting and carrying heavy items"? and are 
utilised for home delivery services!! and for cleaning. Using robots may 
have the advantage that humans no longer endure the poor work condi- 
tions or perform dangerous tasks. Warfare, for example, is becoming 
increasingly outsourced to robots (eg, killer drones) and according to 
several experts, self-driving cars will, in time, be able to drive around more 
safely and in a more energy-friendly way than most humans.!* 

Although the market share of HRs is currently relatively small, 
it is expected to grow exponentially in the coming years, from about 
$1.3 billion in 2022 to about $17.3 billion in 2027.'5 Unsurprisingly, 
ever more companies, non-profit organisations and governmental 


Humanoid Robots 15 


agencies are investing in and experimenting with HRs. Toyota, one of 
the frontrunners, is interestingly approaching HRs as a step beyond 
VR: 


Thanks to VR, there’s no doubt that people have already experienced what 
it feels like to have their movements represented in a virtual world. As an 
extension of this, avatar robots like T-HR3, which possess an actual body, 
are capable of going beyond VR to physically influence the real world. Of 
course, we still have to overcome mountains of development issues, but in 
the future, people will be able to extend their ability to move and experience 
the world using remote avatar robots — and the ability to offer new mobility 
services like this is a goal that is well suited to Toyota as it transforms into a 
mobility company. 6 


Most current HRs are suitable for one preprogramed task. Although 
adaptive to context and circumstances, this usually involves a relative 
demarcated area of expertise. Such HRs are used, inter alia, in the leisure 
industry. For example, the Japanese hotel Henn Na has robots as work- 
force (front desk assistants, cleaning personnel etc), which can take the 
shape of humans, pets and even dinosaurs.!” There are also humanoid 
bartering kiosks, where a HR prepares and serves food and beverages. !® 
Airports are increasingly making use of customer service robots and HR 
luggage carry personnel. Inter alia, HRs can scan passport data, auto- 
matically fill in documents, issue electronic queue tickets and provide 
relevant information to visitors.” In addition, there are HR taxi drivers 
and tour guides at tourist spots all around the world.?° One of the many 
advantages of smart HRs is that they can speak almost any language.*! 
Experiments run with HRs performing various tasks, such as exploring 
the depth of the ocean,” assisting in educational tasks,” packaging and 
delivery” and in industrial environments.” 

For text production and social intersectionality, several chatbots exist, 
the most well known currently being ChatGPT.”° Chat programs allow for 
an advanced combination of a potential friend (social conversations) and 
a virtual assistant that can write a love story in Shakespearian style, look 
up concrete information or be a psychiatrist when needed.” Although 
it is not perfect, it is widely seen as almost fully human-like in terms of 
its conversational abilities. One of the criticisms is that ChatGPT some- 
times makes factual errors, but, arguably, this only makes the bot more 
human. The criticism does uncover an interesting question: do people 
really want a HR or are they looking for a better-than-human compan- 
ion? On the one hand, excessive intelligence and advanced capacities may 
alienate people from robots and arouse feelings of social discomfort; on 
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the other hand, there is a very low tolerance of robotic mistakes. Other 
AT programs exist that can produce video, audio and images.** 

In the medical domain, HRs can be used to perform certain tasks 
that can involve embarrassment, such as matters involving genitalia 
or sexual diseases. Men and women who do not want to be exam- 
ined or treated by a doctor of the opposite sex for religious or other 
motives may have less trouble with a gender-neutral robot or HR that 
can change its physical appearance according to the patient’s prefer- 
ences. HRs are also deployed to assist with non-patient-facing tasks, 
including running patient supplies, delivering lab samples, fetching 
items from central supply, distributing protective equipment and deliv- 
ering medication. There are HRs that autonomously disinfect rooms, 
with one of the many advantages being that HRs cannot get infected 
(eg, by COVID-19).*? Robots are used for home and elderly care. 
These robots are often multi-functional. They can help with cleaning 
the house, helping people to take their medicine in time, with getting 
up or supporting them when walking, dressing and undressing; they 
can also help people with cognitive problems to organise their lives, 
and they can act as a friend and conversational partner. In addition, 
home care HRs may be equipped with diagnostic software to allow for 
early diagnoses of, for example, amyotrophic lateral sclerosis (ALS) or 
Alzheimer’s disease.* 

Government organisations also use HRs for a wide variety of tasks. 
In warfare, having HRs as foot soldiers might have the advantage that 
the opposing army believe they are facing human soldiers, while in fact 
they are better equipped and more able than average human soldiers, 
as well as being better shielded against the impact of bullets and other 
ammunition. HRs are used for friendly patrols, for example, at shop- 
ping centres. They can be designed in a way that their appearance does 
not stand out, and they look small and unintimidating, while being as 
strong as or stronger than most human security guards.*! The Russians 
make use of HRs in space’? and the US’s National Aeronautics and Space 
Administration (NASA) is increasingly investing in robonauts: 


Working side by side with humans, or going where the risks are too great for 
people, Robonauts will expand our ability for construction and discovery. 
Central to that effort is a capability we call dexterous manipulation, embod- 
ied by an ability to use one’s hand to do work, and our challenge has been to 
build machines with dexterity that exceeds that of a suited astronaut.” 


Some HRs are designed for their own sake, without a concrete use case 
in mind, and many others for social interaction.*+ For example, Hanson 
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Figure 2.1 Patient care robot?’ 
Source: ZUMA Press, Inc/Alamy Stock Photo. 


Robotics won a United Nations (UN) innovation prize with its robot 
Sophia. Hanson believes that HRs with good aesthetic design, rich 
personalities and social cognitive intelligence can potentially connect 
deeply and meaningfully with humans. It has updated its model to 
Sophia 2020, which has several functionalities, including customised 
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skin tone, facial design, language and arm colours, integrated human- 
like emotional expressions, sentence and context understanding, mouth, 
face and whole body motion synchronised with speech output, facial 
recognition, body tracking and 74 degrees of motion freedom.” Pet-like 
robots also exist.” The purpose of these is to act as a canine friend,°* to 
execute inspections?’ or to make announcements. One such pet robot 
is Moxie, which is designed to be an AI friend to children and to help 
them grow intellectually and personally (‘What do you get when advisors 
from Walt Disney and NASA team up with education experts? You get 
Moxie’).*! The team is quite ambitious in terms of what interaction with 
Moxie can accomplish in relation to a child’s wellbeing. Moxie is said 
to help children, inter alia, with emotion regulation, relationship skills, 
expressing kindness, conflict resolution, acquiring calming techniques, 
understanding social cues, building healthy habits, showing leadership, 
acquiring communication skills, mindfulness, self-confidence, showing 
empathy and supporting others.“ 

A final area of application worth mentioning is HRs used as sex 
or love robots.*? These robot functionalities are so popular that some 
experts predict that by 2050, having relationships with a robot might be 
fully normalised and socially accepted. Others even claim that by that 
time, more sexual intercourse with sex robots will take place than with 
conspecifics:* synthetic love. Sex robots can cater to romantic needs 
and to sexual desires; sex doll brothels have already opened in Asia.*® 
One of the advantages is that sex work, which can be seen as degrading 
and exploitative by nature, is no longer performed by real women. Love 
dolls can also be helpful to people who have suffered from traumatic 
experiences? and help elderly people or people with mental or physical 
disability to have a healthy sex life.*® 


However, there are also a number of potential disadvantages to the 
use of HRs.” To start with the last field of application discussed above, 
experts have pointed out that sex with HRs is mostly performed by men 
and that the use of a sex doll may feed into a macho-male perception of 
women and sex, as the robots are usually designed to obediently cater to 
the needs of their owner and to always be ready to perform their sexual 
duty, without having desires or preferences of their own. Some suggest 
that this might normalise a modern form of slavery and sexual exploi- 
tation. Either men copy that behaviour to the real world, with all the 
problems that go with it, or they prefer to stay with their sex robot.°° 

In addition, a fear of robotics in general is that of privacy threats, as 
these robots are equipped with multiple sensors, and of hackability. Since 
robots are often strong and sometimes work with vulnerable people, they 
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can do substantial harm, which can be as small as being reprogrammed 
to forget to remind an elderly person to take their medicine. There are 
questions about the extent to which every aspect of human interaction 
can be programmed or substituted by AI. For example, medical experts 
often operate on what they see as intuition: 


Intuition is often developed through experiences, non-analytical reason- 
ing and thinking, feelings, a solid knowledge base and non-linear creation 
of knowledge. In patient care, intuition is very valuable and is often used 
because when needed, it re-emerges in the form of ‘intuitive intelligence’ or 
‘gut feelings’. However, because intuition is very difficult to capture or quan- 
tify, and thus stored so it can be later uncovered or mined as with other [big 
healthcare data] BHD; intuitive knowledge is often regarded as non-scientific 
and not suitable in investigations in the sciences including healthcare. This is 
in contrast to evidence-based medicine and decision making based on quan- 
titative information from patients’ data and improved health outcomes that 
can be measured and stored as BHD for later use.’! 


In addition, some experts believe that AI may not be good at picking up 
signals of non-identified diseases or in experimental diagnostics. 

Another general fear of AI is that it is biased,’ that humans might 
trick the self-learning AI system by giving selected input ques,” and 
that most self-learning systems are wholly or partially black boxes. This 
might reduce social acceptance of HRs and lead to unexpected outcomes. 
The lack of trust in HRs may be enhanced by the lack of universal and 
generally accepted standards for safety, audits, interoperability and risk 
management.‘ HRs are mostly produced, programmed and operated by 
private organisations with a commercial goal, which means that profit 
may be a bigger driver in programming HRs’ algorithmic behaviour 
than the general interest or the interest of humans with which they inter- 
act. Finally, there are concerns in relation to powerful Large Language 
Models such as ChatGPT in terms of their ability to write essays and 
assignments. As a result, many universities are faced with the question of 
whether to prohibit the use of this program by students and, if so, how to 
enforce such a prohibition.” 


2.3. DEEPFAKES 


Deepfakes (DFs) allow citizens and companies to generate funny videos 
and satirical memes.°® An example is putting Nicolas Cage into even 
more movies than he has already appeared in,” putting two politicians 
from opposing parties on Temptation Island’? and having the head 
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of state deliver an alternative Christmas speech.*? In a more homely 
setting, children can produce a funny video of their parents, while a 
fake video can be produced in which a celebrity appears to congratulate 
the birthday girl or boy or a DF of George W Bush talking about the 
Axis of Evil may be so altered that the name of the neighbour appears 
instead of Iran. 

DFs also allow dead people to be brought back to life. Salvador Dali 
may give a virtual tour in a museum” or Mona Lisa may address viewers 
in person.*! Both Google™ and Microsoft® allow a person to enhance 
old photos and make them more realistic. This technology may be used 
to bring an ancient ancestor back to life: seeing and hearing great- 
grandmother walk and talk in what appears to be authentic video mate- 
rial. Deceased loved ones can also be brought back to life through DFs, 
for example to give a speech at their own funeral, or to communicate 
with a late spouse to keep in contact. 


‘It makes me so happy to see him smile again’, one user said after animating 
a photo of her husband, who died 4 years earlier. ‘It’s as if they are looking at 
you and your surroundings and seeing how much things have changed’, said 
another user. Reporter Joe Fitzgerald Rodriguez commented that the feature 
gave him a chance to see his late father’s face move again after he lost the only 
videotape he had of him years ago. ‘Forget iPhones and self-driving cars’, one 
commenter said in response to a Deep Nostalgia™ animation. “This is the 
moment we officially started living in the future!’ 


Grief therapy by communicating with a deceased spouse is an option as 
well.®© 

DFs are also increasingly being used in the film industry. In the film 
Furious 7, Paul Walker, who died while still shooting, was seemingly 
brought back to life to complete the film.®® The technology also allows 
an actor to be scanned to have their virtual image star in films, saving 
time, money and energy. This may be particularly advantageous for work 
now performed by stuntmen.®” DFs have a broader effect on the enter- 
tainment industry as well. For example, virtual influencers and bands, 
which are not based on any existing person, have been created and have 
already acquired a large following and sponsorship contracts and have 
given performances in physical concert halls.6* DF technology can also 
be used for social interaction between citizens, in both personal and 
business contexts. For instance, in live video calls, people can assume 
the appearance of a cat, dog or fictional character,” and if participants 
speak different languages, their words can be translated live while their 
lips are synchronised to match the translation. Thus, a Chinese person 
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and a Danish person can communicate with each other in their own 
respective languages, while the other perceives their conversation partner 
as speaking in their language. According to researchers, this technique 
will ‘significantly improve the overall user experience for consuming and 
interacting with multimodal content across languages’.”” 

DF technology is also used for a wide variety of purposes in the 
medical domain.”! For instance, synthetic Magnetic Resonance Imaging 
(MRI) images of brains with tumours can be created and algorithms can 
be trained from these images to detect early forms of cancer or brain 
diseases, such as Alzheimer’s. Not only can this technology be used for 
MRI scans, it is also capable of developing images of liver lesions”? and 
skin lesions that can be used by dermatologists.” In addition, through 
DF technology, people with ALS could retain the ability to speak using 
their own voice,’”* which is also the case for people with dysarthria, a 
speech disorder caused by damage to the nervous system.” The privacy 
of citizens could also be enhanced or protected. For example, by having 
the patient assume a different identity, they could remain anonymous 
while being able to participate in live conversations, as a witness in a 
criminal trial or as a patient in a clinical trial, for example.” 

DF technology also offers opportunities within the context of law 
enforcement. Virtual people can be used to infiltrate criminal networks” 
and Sweetie, a virtual avatar, was deployed to track down child molesters 
and pederasts: 


Sweetie is a ten-year-old virtual Filipino girl. A very realistic girl deployed 
online in chat rooms and dating sites. When men approach her for sexually 
oriented chats, she engages in conversation with them. All the information 
this yields is stored and used to warn, track down, or even catch and convict 


perpetrators.” 


Sweetie 2.0 has led to convictions in multiple jurisdictions.” More contro- 
versially, DF child pornography can be produced starring non-existent 
children to infiltrate networks or track consumers of that content. 

Retail companies can develop virtual models which customers can 
use to swap their own bodies for those of virtual models. This will make 
it possible to virtually try on clothes or glasses by taking the custom- 
er’s body type into account. The app Superpersonal offers a virtual 
fitting room, allowing customers to try on clothes based on data about 
their gender, height and weight.5t The demand for e-commerce has 
increased due to the closure of physical shops globally during and after 
the pandemic. The size of the virtual fitting room market is expected to 
grow from over US$3 billion in 2019 to $6.5 billion in 2025.87 
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Within the educational context, DFs offer the opportunity to deliver 
more interactive lessons, for example, by a history teacher who can 
have Napoleon speak in class about which decisions he made and 
why.®? Charities deploy DFs by having a celebrity call out to support a 
good cause in all the world’s languages,** by having a politician make 
a statement that motivates people to act® or by showing people who 
do not support providing shelter to refugees what a war ruined city 
looks like by turning their own village into one. Using DF technology 
also allows politicians to reach out to minority groups in their own 
language.*” 


There are at least an equal number of applications of DF technol- 
ogy with negative consequences. One of the first DFs was produced and 
shared by citizens in 2017 on Reddit: faces of famous people such as 
Taylor Swift were placed on the bodies of porn actors. This practice 
has remained popular since then. From February 2018, major platforms 
responded by restricting the applications of DFs,** including Reddit.” 
The vast majority of DF porn is of women: the women tend to be either 
celebrities or people in direct contact with the producer of the DE, such 
as classmates, colleagues, neighbours or exes.” While female politi- 
cians are regular victims of pornographic DFs, male politicians are often 
Deepfaked in a way that they appear to use foul language or make contro- 
versial claims,”! such as Donald Trump supposedly calling on Belgium to 
get out of the Paris Agreement” or Barack Obama appearing to make 
Trump out to be a ‘total and complete dipshit’. 

In April 2020, DF audio clips surfaced using a text-to-speech model 
trained in Jay-Z’s speech patterns to impersonate him quoting the “To be, 
or not to be’ monologue from Shakespeare’s Hamlet.*+ Jay-Z’s label peti- 
tioned for its removal due to copyright infringement. YouTube initially 
responded by removing the video, but later reinstated it online because 
the plaintiff had not provided sufficient grounds to show that the mate- 
rial was indeed unlawful.” The use of data relating to dead people also 
raises the question of the extent to which intellectual property rights rest 
on images of dead people and the extent to which their descendants can 
invoke them. Increased exploitation of the deceased’s imagery might also 
trigger a debate on post-mortem privacy: should deceased people have 
reputational and personality rights?” 

DFs can also be used for financial gain, such as manipulating 
markets. After a false message circulated on WhatsApp in 2019 stating 
that Metro Bank was no longer liquid, people flocked to bank branches 
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to claim their money. This ultimately led to the company’s stock falling 
by 9 per cent.” Criminals can use DFs to impersonate chief executive 
officers (CEOs) of listed companies, appearing to make statements that 
cause share prices of their companies to fall. Europol therefore calls DFs 
a considerable danger as they might be used for, among other things, 
‘perpetrating extortion and fraud, facilitating document fraud, falsifying 
online identities and fooling KYC [Know Your Customer] mechanisms, 
falsifying or manipulating electronic evidence for criminal justice investi- 
gations, disrupting financial markets’ and, for example, extracting trade 
secrets through DFs.’ An early example of fraud comes from 2019, when 
someone was defrauded by means of a DF audio fragment. A CEO of a 
UK energy company transferred €220,000 to a Hungarian bank account, 
believing he was on the phone to his boss, the head of the German parent 
company, who seemed to be instructing him to do so.” 

By means of a DF video, a politician during a general election could 
discredit their opponent or stir up a political scandal. For example, a DF 
of former Italian Prime Minister Matteo Renzi, in which he insulted a 
fellow politician, sparked public outrage. 


The deepfake video refers to Renzi’s decision Sept. 17 to leave the Democratic 
Party and form his own party. In the parody, the supposed Renzi is seen 
talking when he thinks he is off air. He discusses the reaction of various 
politicians, including Prime Minister Giuseppe Conte; Luigi Di Maio, leader 
of the Five Star Movement; and Italy’s president, Sergio Mattarella ... The 
video is so outrageous that it is clearly a parody, but deepfake technology 
makes it look incredibly realistic. So when people started sharing it online, 
claiming that it was a real video, quite a few social media users fell for it and 
were outraged by what they saw as Renzi’s bad behavior.! 


As such, DFs could influence democratic elections, a threat that 
could come from domestic sources and from foreign nations such as 
Russia.!°! For example, in a letter to the House of Representatives, the 
Minister of the Interior of the Netherlands pointed to the dangers of 
disinformation: 


[P]olitical influence is not a new phenomenon. It comprises the integral, 
often covert use of (misleading) arguments, selective information and 
disinformation (regarding politically sensitive themes) for the purpose of 
achieving political goals towards a predetermined audience. However, the 
emergence of the internet has triggered a new dynamic: wide dissemination 
of disinformation and fake news is easy, quick and cheap to generate and 
often, the source can remain anonymous. As indicated in the annual reports 
of the intelligence agency, there are state actors that focus on the Netherlands 
and that have the intention and capacity to interfere in democratic processes. 
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They are interested in political decision-making and high-quality techno- 
logical knowledge (ICT, maritime technology, biotechnology and aerospace 
sectors). 10? 


DFs can also be used to provoke domestic or interstate conflict. In 2018, 
Gabon’s President Ali Bongo was out of the country for medical treat- 
ment. People grew suspicious of the President’s wellbeing, with some 
believing that he had already passed away. After several months, the 
Vice-President announced that the President had suffered a stroke. Civil 
society and the general public continued to question why the President 
had not yet appeared in public, prompting the government to release a 
video of him. Many did not believe the video was real and a week after 
the video was published, the Gabon army staged a coup, which ulti- 
mately failed. 10 

The relationship between different states can also be strained by 
DFs. For example, in 2020, Australian Prime Minister Scott Morrison 
demanded an apology from China after a Chinese Foreign Ministry 
spokesman posted a fake photo on Twitter (now known as X) showing 
an Australian soldier holding a knife to the throat of an Afghan child; 
however, China refused to do so.! Europol also fears that DFs will be 
used for ‘distributing disinformation and manipulating public opinion, 
inciting acts of violence toward minority groups, supporting the narra- 
tives of extremist or even terrorist groups, and, stoking social unrest and 
political polarization’.!° For example, in Myanmar, DFs of members of 
the Muslim minority were created and spread, showing them to perform 
controversial activities in order to incite hatred and violence against them. 


2.4. AUGMENTED REALITY 


One of the first applications of Augmented Reality (AR) was in the 
movie industry: a great many developments have materialised since, such 
as turning 2D images into 3D ones by adding layers of information to the 
film or by stimulating other senses than sight, such as producing olfac- 
tory and haptic sensory input correlating with the visual cues. In sports, 
a person’s performance rates, biophysical indicators and other relevant 
information can be visually projected through an AR device, potentially 
comparing that information to that of competitors. It is also used for 
sport fans watching a game — for example, having AR project the offside 
line on the TV coverage of a football match or replacing the puck in 
ice hockey for a special coloured symbol that allows viewers to more 
easily follow the game. In gaming, old-fashioned board games may 
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be turned into 3D environments when viewed through an AR device; if 
interconnected, the AR devices of the players who are physically remote 
may project the same 3D environment onto different 2D game boards 
and allow for remote interaction. Alternatively, a game master can use 
AR to create a virtual environment populated with avatars, tying them 
to a specific location in the real world where the players are. The most 
famous of all AR games is perhaps Pokemon Go, in which figures and 
items are projected onto the real world, which participants need to catch 
or interact with. 10 

The first popularisation of AR was arguably by Boeing, who had 
employees wear goggles when working on machines to allow them to 
receive additional information. Today, AR is used by the military, in 
aviation, in car factories and many other production fields. Inter alia, by 
using goggles, mechanics can see each step necessary for repair or assem- 
bly, identify the right equipment and scroll through textual instructions. !” 
For repetitive precision work, artificial stimuli can provide warnings if 
something goes amiss, resulting in more reliable production processes. 
AR is also increasingly used in commercial settings. For example, an 
app may show ratings, reviews or advertising when customers shop in 
commercial areas.!!? AR can be used to virtually try on clothes, allowing 
the user to see how they would look by looking in the mirror through an 
AR device." 

In social settings, AR can be used to facilitate interactions between 
humans and avatars, such as through virtual avatars also being invited 
to a party and to join in on discussions. AR can be used to scan faces 
using facial recognition and provide overlay information — for example, 
from LinkedIn or other webpages. Emotion recognition can be used if, 
for example, a neighbour rings the doorbell aggressively. Wikitude!!* 
allows users to project information from Wikipedia onto reality.!™ In a 
similar vein, AR can be used in tourist settings, such as by overlaying the 
visual representation of the Colosseum in Rome on the ruins that still 
stand.!!$ A virtual tour guide can inform the visitor about the histori- 
cal sites, adapting to contextual cues and the specific questions of the 
visitor(s). Similarly, museum experiences can be enhanced using AR. For 
example, in a natural history museum, dinosaurs may be brought to life 
visually, or their assumed sound and smell can be produced. For foreign 
tourists in particular, AR could prove vital when signs in local languages 
are incomprehensible. AR is also used to obscure things, such as military 
infrastructure or the wall that separates Israel from Palestine.!'6 

In the context of security, there are several AR apps that allow citi- 
zens to collaborate with law enforcement officials to solve crimes in a 
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game-like setting!” and to get under the skin of a border patrol agent, a 
police officer patrolling the streets or a combatant in an armed overseas 
conflict. Chinese police officials use smart glasses to identify potential 
suspects.!!8 Police officers may use emotion recognition to obtain infor- 
mation about the emotional state of a suspect. The US military uses AR 
for military training: 


As the Army looks to further modernize its force, it has focused on synthetic 
training environments to boost Soldier readiness and lethality. IVAS is a 
single platform that Soldiers and Marines use to fight, rehearse, and train. 
The Soldier Lethality Cross-Functional Team identified technology that 
makes it possible to deliver a single system across the force. Similar technol- 
ogy has been used by the Navy and Air Force, which have used it to train 
fighter pilots. 


AR is used in medical settings as well, inter alia, allowing medical 
professionals to produce 3D virtual models for analysis and surgical 
planning.!”° The visualisation of the path through the anatomy of the 
area where a tumour must be removed, for example, can be done by first 
creating a 3D model from multiple views and slices in the pre-operative 
study. The model can be projected over the target surface to help the 
surgical procedure. Developers have created an AR contact lens to meas- 
ure blood glucose and cholesterol, and to project relevant information on 
a patient’s body while a surgical team is performing a procedure. Using 
an optical see-through display can also help ultrasound technicians by 
allowing them to view volumetric rendered images of foetuses overlaid 
onto the abdomen of pregnant women; this image can depict what is 
occurring inside the abdomen."! 

An important area of application for AR is in education. AR educa- 
tional tools have become increasingly popular, with the COVID-19 
pandemic leading to a particular rise in this popularity on account of 
the increase in home schooling. AR can assist students with their tasks 
and give immediate feedback when performing them. Research suggests 
that using AR, difficult or tough topics can be gamified, thus making 
learning more fun and thereby increasing students’ performance ratings. 
Through AR, students can also interact with and challenge each other — 
for example, they can compete to win a gold medal in quizzes or tasks. 
Biology students can study virtual organs while being together in a 
physical classroom. History lessons may be partially taught through 
role-playing games, where children can wear their normal clothes, but 
AR turns their outfit into one of the historical figures. Augmented 
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books can be designed, with symbols, historical figures and buildings 
popping up next to the relevant text or with additional information 
being shown next to key words, figures or events. The goal is to make 
reading more fun for young and adolescent readers. Some devices allow 
an overlay of astrological information on or connecting lines of a zodiac 
sign projected on the sky at night, which might be used for astrological 
training.” 

A final positive way in which AR is used is in smart environments, 
such as smart homes, smart cities, smart cars and navigation apps.!*4 
In smart homes, disabled people can control the environment using a 
handheld mobile device equipped with a Mobile AR (MAR) application: 


The activity could be operating electrical appliances, opening the doors, 
switching off or dimming the lights. Increasing the accessibility of people 
having motor disabilities in a smart city enables the users to interact with 
items placed beyond their arm’s length. Navigation through smart cities 
using MAR provides visitors with a better experience. MAR enhanced smart 
building management systems are capable of recognizing building geom- 
etry, simulate building visualisation, identify assets and incorporate the 
feedback from the user to enable proper management of the smart build- 
ing. Applications like IKEA Place equipped with AR, allow users to visualize 
the furniture placement before making the purchase decision ... For smart 
homes, security monitoring enabled with AR can alert suspicious behav- 
iors and current warnings on property owner’s remote mobile screen. These 
warnings can be enhanced with the details of suspicious individuals if they 
have criminal records. !*5 


The French Agency for Food, Environmental and Occupational Health 
& Safety has summarised the dangers of AR and VR as signalled in 
the academic literature, although not all these effects have been proven 
beyond doubt:!?6 


1. Effects relating to the ergonomics of AR/VR interfaces (muscu- 
loskeletal disorders, accidents, effects due to the hygiene of the 
interface and to noise levels). 

2. Cybersickness: nausea, vomiting, headaches, general discomfort, 

vision effects, physiological effects (cardiac, gastric, respiratory, 

dermal etc) and vestibular effects (dizziness). 

Impaired sensorimotor coordination caused by exposure. 

4. Psychological and psychosocial effects such as emotional risks, dere- 
alisation (detachment from one’s surroundings), dependency on the 
interface and content, content-related effects (violence, relationship 
to sexuality etc) and social isolation. 


ies) 
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5. Effects that alter self-image (through avatars). 

6. Effects relating to physical agents emitted by virtual and/or 
augmented reality devices (for example, blue light and temporal 
light modulation, and electromagnetic fields). 

7. Neurological effects (epileptic seizures). 

8. Effects on development (emotional and cognitive) of the visual and 
auditory systems. 


Of these effects, perhaps most attention has been paid to the safety risks 
created by Pokemon Go, as children crossed busy streets while playing 
the game without paying attention to traffic. A more underwhelming 
criticism of AR is its relative lack of usefulness: ‘AR is fun for gaming 
but it has no actual purpose. None could find a reason to pay so much 
money for a device which has no useful purpose.’! A scenario that is 
real but relatively unexplored is where a hacker deliberately places incor- 
rect information in the AR, leading people to make wrong decisions that 
negatively affect them, others or the environment. Even without mali- 
cious intent, AR can have harmful effects. The most frequently used 
variants of AR have one thing in common: additional information being 
provided to the user. This may lead to information overload or distract 
the user when they need to focus, such as in traffic. #8 AR-generated texts 
or objects may block important cues from sight. Even the practice of 
projecting additional textual information or infographics onto the sides 
of the user’s view may be dangerous, as objects in the periphery of one’s 
sight may provide important information. 


Peripheral vision is more important than you might think, because it provides 
a wealth of information about speed and distance from objects. Central vision, 
despite the great detail it offers, gives you only a rough estimate of movement 
toward or away from you, based on changes in size or in the parallax angle 
between your eyes. But objects moving within your peripheral vision stimulate 
photoreceptors from the center of the retina to the edge, providing much better 
information about the speed of motion. Your brain detects objects in your 
peripheral field and evaluates if and how they (or you) are moving. Interfering 
with this process can cause you to misjudge relative motion and could cause 
you to stumble; it might even get you hit by a car one day. It’s ironic, really. You 
buy an AR device to make you more able, yet you’re likely to experience some 
of the same problems faced by visually impaired individuals: reduced depth of 
focus, distance and speed perception, and reaction time. Indeed, AR users may 
be at more risk than someone with a permanent vision problem, because they 
have developed no compensatory strategies for lost vision.!?? 


Gaming is one of the most prominent modern addictions among chil- 
dren, which might be incentivised using AR.'°° A negative consequence 
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of AR devices is their effect on the privacy of both the AR user and 
of bystanders,'3! as AR devices that are equipped with facial and 
emotion recognition software gather substantial amounts of sensitive 
data about people that generally have not given their consent for this 
to happen.!* Social discomfort may also arise when different people 
experience different realities — for example, at a party, some see and 
hear avatars while others do not. AR can also make certain tasks more 
‘fun’ or less ‘appalling’, such as making warfare less bloody, and this 
may withhold important negative sensory input from the combatant, 
which normally leads to reticence. AR can also be used to block certain 
‘unnecessary information’, which may include innocent citizens being 
present near a drone strike target. 


2.5. VIRTUAL REALITY 


Various applications have emerged for Virtual Reality (VR). Most of 
these align with the use cases for AR, although often taking the concepts 
one step further. For example, especially during the COVID-19 pandemic, 
VR was propagated as facilitating a new form of tourism: tourism 2.0. 
VR allows citizens to roam the world and experience spectacular sites 
while remaining in the comfort of their chairs. This form of travelling 
has a positive effect on the environment because it lowers the carbon 
footprint of travelling. VR can also allow people to experience specific 
sites that are threatened by climate change without aggravating the prob- 
lem by producing waste production or relying on tourist infrastructure, 
which could be particularly valuable for areas such as the polar regions. 
There is also room for VR-enabled crisis tourism, where people want to 
go to a site where a catastrophe has occurred or to a war zone. This may 
cater to voyeuristic needs and may also cause people to donate to chari- 
ties and good causes after having witnessed the impact of catastrophes 
with their own eyes. "4 

VR is also used in medicine.!*° It is used in implantology and 
orthognathic surgery, inter alia, to reconstruct the orbital floor follow- 
ing blowout fractures and the planning of mandibular reconstruction 
following cancer resection. During the COVID-19 pandemic, VR was 
used for staff training, patient education and virtual consultations. !%° 
VR treatments are being experimented with in relation to patients with 
somatoparaphrenia (a delusional perception that a limb or side of their 
body does not belong to them). It is also used for pain relief. Because 
people are prone to identify with the avatar, their healthy representation 
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in VR may relieve pain or diminish the feeling of disembodiment. 
Importantly, these effects are not temporary, but may have long-lasting 
effects after the user has returned to Physical Reality: they may even 
be permanent because the brain circuits can be partially reconfigured 
through VR.!%” For the visually impaired, training in a virtual environ- 
ment can help while learning to navigate using cane skills. 138 

VR is considered especially promising for the treatment of psycho- 
logical disorders!’ and sexual therapy,“ although there are relatively 
few studies on the concrete effects of VR in treatment.'4! VR is used 
for the treatment of anxiety disorders, such as phobias, post-traumatic 
stress disorder, panic disorder and agoraphobia, social anxiety disor- 
ders, psychological stress and generalised anxiety disorders. It is also 
deployed for the treatment of eating and weight disorders, because VR 
exposure to stimuli, such as food, may reduce the level of anxiety elicited 
by them in real life, because it may allow for the reconsolidation of nega- 
tive memories and because it may change a person’s body representation 
or perception. For example, an obese person’s VR representation may 
be slim, so they can get accustomed to perceiving themselves as being 
thinner. VR is also experimented with in the treatment of children with 
autism spectrum disorder and neurodevelopmental disorders, such as 
attention-deficit hyperactivity disorder (ADHD),!* and for people with 
dementia, to bring them back to a happy place in their youth. Moreover, 
VR-based cognitive neurorehabilitation systems are used: 


[Flor improving the rehabilitation of stroke patients with arm and hand pare- 
sis. Using a custom, low-cost kinematic tracking system designed for clinical 
or home use, patients engage in task-oriented interactions with objects in a 
virtual environment. Our paradigm is based on the hypothesis that observed 
actions correlated with self-generated or intended actions activate the motor 


pathways by means of the so-called ‘mirror-system’.!*7 


One of the domains in which VR has truly thrived is in education. 1“ 


There are a myriad of applications, including bringing schoolchildren 
back to ancient sites during history classes. In biological and medical 
sciences, VR equipment can have a significant advantage over conven- 
tional education, both because material may not be available (eg, due 
to a limit in the number of human or animal organs or body tissue) and 
because it may be difficult for a whole group of students to see the actual 
material and the dissection of it.!* In medical training, considerable 
investments have been made in screen-based learning, 360° visualisation 
and interactive VR rooms where students can work together.'*© Most of 
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the existing models of simulation focus on teaching surgical trainees’ 
technical skills. 147 
Mixed reality (MxR) is used for military training as well: 


The Close-Range Weapons Simulators (CRWS) project provides a good 
example of how real equipment — in this case deactivated weapons removed 
from the original shore-based training establishment of HMS Cambridge — 
were used to augment the VR experience. Observations of training proce- 
dures at HMS Cambridge drove the choice of a head-tracked [head-mounted 
display] HMD-based solution for the VR system, based on the interactions 
of gunners with their weapons and other shipboard personnel. However, the 
techniques by which the weapons were physically moved during operation 
dictated that an MxR training solution was essential. To operate the 20 mm 
cannon, the gunner is strapped into the shoulder rests and uses his full body 
weight in order to move the weapon in azimuth and elevation. For the 30 mm 
cannon, the gunner sits within a small open cabin and operates the azimuth, 
elevation and firing functions of the weapon via a small control panel. 
These features also drove the choice of a partially face-enclosing HMD, the 
Kaiser XL-50, which, at the time, afforded a degree of unobscured real-world 
peripheral vision to the wearer.!*8 


VR is also used in the film industry. Combining 360° film material with 
VR-enabled interactivity, immersion and narratives is sometimes called 
‘total art’. Preliminary experiments with these types of films show 
changes to the viewer’s perception from ‘merely’ viewing a movie to 
experiencing it — living the film. Especially when shot from a first-person 
perspective and when the film allows the viewer to give interactive feed- 
back, these experiences may be perceived as real.'4? Total art may blur 
the line between gaming and watching a movie, fitting with the trend of 
producing games that have film-quality audio and video as well as an 
increased emphasis on narrative. Interestingly, the media is also start- 
ing to embrace the potential of VR for news coverage — for example, for 
covering conflict zones. 10 

Another flourishing field of application, as with HRs, is the use 
of VR for sex-related purposes. VR pornography is appreciated more 
by its viewers than 2D pornography. It gives the viewer the feeling of 
intimacy and presence, allowing for more interaction and a stronger 
first-person experience, as well as increased sexual arousal.!>! Even in 
2017, Pornhub reported that VR porn was one of the fastest-growing 
segments in the porn industry.!°? The advantage is that no porn actors 
need to be involved, so certain explicit or humiliating activities need 
not be performed by humans.!°3 One sub-branch of VR porn is that of 
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teledildonics, which is a fully immersive, mixed-reality experience that 
combines VR porn with physical objects and stimulators, such as dildos, 
vibrators and plugs. This technology can also be used by partners who 
live apart from each other to enjoy a realistic romantic sex experience.!+ 


Many of the dangers of VR or dangerous applications have already 
been touched on while discussing the negative use cases of AI robotics 
and AR, including privacy and security risks, and the risk that a hacker 
may infiltrate the virtual world.!* Because people are more immersed in 
VR than in AR, a malicious hack could have an even more severe impact, 
especially when people use VR for the treatment of traumatic experi- 
ences. As most VR worlds are not designed exclusively by the user or 
the company offering the VR world, co-users may affect the design of 
the VR world and the activities that take place therein. This means, for 
example, that users may be confronted with other avatars performing 
explicit activities, which may trigger old traumas or create new ones. 16 
Alternatively, a VR avatar may be used to impersonate a person against 
their volition, a virtual form of identity theft that may both lead to atroc- 
ities in the VR world itself as in the real world, as viewing a person’s 
avatar performing certain activities or making certain statements may 
have either a conscious or subconscious effect on how other users see 
them in real life.” 

Since VR worlds are based on as-if scenarios and because they often 
allow users to start over or have different lives in different worlds, this 
might cause users to become estranged from real-life scenarios, where 
decisions can often not be reversed, and it is not always possible to start 
over again. Another potential effect of VR is that experiences are ‘opti- 
mised’, so people might have difficulty going back to non-VR-mediated 
experiences, such as with sex. Certain VR games, such as Dead or Alive 
Xtreme 3, allow the user to sexually assault women. Another fear is 
that of body neglect, especially when adolescents stay in VR worlds for 
prolonged periods, preferring their avatars and the VR world to their own 
bodies and lives.!5* This may lead to self-loathing, distorted self-image, 
depersonalisation and derealisation, giving rise to mental disorders as 
mentioned in the Diagnostic and Statistical Manual of Mental Disorders, 
Fifth Edition . 

In addition, using VR may have physical effects. A substan- 
tial number of users suffer from motion sickness.©? This is 
especially prevalent in VR situations where a person is supposed 
to walk in order to roam the virtual world. The eyes of a user can 
become fatigued when looking at virtual screens for a long period 
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of time. In VR games too, users tend to push their limits beyond 
what they would naturally do — for example, in terms of sleep.'®° 
Importantly, the long-term effects of VR experiences are largely 
unknown, and data about the impact on children that partially grow 
up in such worlds will most likely only be available in 20 years’ 
time. Among other predictions, experts suggest that staying in VR 
for prolonged periods of time may lead to social isolation. Also, the 
fact that users each live in their own worlds may lead to a decrease 
in shared experiences, which is the cement of social interaction and 
a well-functioning society. However, experts also point out that 
impersonating others through VR may lead to greater empathy and 
understanding. !*! 


2.6. CONCLUSION 


HRs, DFs, AR and VR have a wide variety of applications. This chapter 
has provided a non-exhaustive list of examples of the most prominent 
use cases and domains in which the techniques are deployed. These can 
be roughly represented as follows. 


surgery 


VR: pain, sex, 
disorder 
treatment 


models 


Figure 2.2 Use of HR, DF, AR and VR in the medical sector 
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Figure 2.3 Use of HR, DF, AR and VR for industry and commerce 
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Figure 2.4 Use of HR, DF, AR and VR for education and communication 
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Figure 2.5 Use of HR, DF, AR and VR in the entertainment sector 
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Figure 2.6 Use of HR, DF, AR and VR in the security sector 
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Several existing and potential future risks and benefits of the four technol- 
ogies have also been discussed in this chapter. These can be summarised 


as follows: 
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Figure 2.7 Benefits of HR, DF, AR and VR 
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Figure 2.8 Risks of HR, DF, AR and VR 


However, it should be stressed that it is not always possible to distinguish 
between positive and negative consequences. The normative evaluation 
of the use of the four technologies depends on one’s personal, ethical and 
political views. Is having Napoleon teach history classes to children good 
in terms of engagement or bad because it accustoms them to living in a 
post-truth society? Is a politician speaking in the dialect or language of 
every minority in their country good in terms of inclusion or a modern 
form of voter deception? Is being able to speak to a deceased partner good 
for one’s mental health or bad because the normal path for grief process- 
ing is disrupted? Is it good that the police use child avatars and fake child 
pornography to infiltrate paedophile networks or bad because it actually 
produces (synthetic) child pornography itself? Is having robots replace sex 
workers good because women are no longer forced to do degrading work 
or bad because it deprives them of work, allows men to develop dark 
fantasies and facilitates them in terms of living these fantasies out? 


3 
Under the Hood 


Architecture and Design of 
Synthetic Technologies 


Al refers to systems that display intelligent behaviour by analysing their envi- 
ronment and taking action — with some degree of autonomy — to achieve 
specific goals. Since AI refers to so many techniques and contexts, greater 
precision is required in order to hold meaningful and constructive debates 
about it. For example, arguments about simple ‘expert systems’ used in advi- 
sory roles need to be distinguished those from those concerning complex 
datadriven algorithms that automatically implement decisions about indi- 
viduals. Similarly, it is important to distinguish arguments about speculative 
future developments that may never occur from those about current Al that 
already affects society today.' 


Defining the precise object of regulation in dynamic technological domains 
is a challenge in itself. Given that Al is still an open-ended notion that refers 
to a very wide range of products and applications, there is no transnational 
agreement on a commonly accepted working definition, neither at the techni- 
cal nor the legal/policy level. As there is no legal and political consensus over 
what Al is, a plurality of definitions has emerged in Europe and worldwide 
that are either too inclusive or too sectorspecific. This fragmented conceptual 
landscape may prevent the immediate development of a lex robotica and 
possibly undermine all efforts to create a common legal nomenclature, which 
is particularly instrumental for the drafting, adoption and effective imple- 
mentation of binding legal norms.” 


European Parliament, European Union 
3.1. INTRODUCTION 


FTER HAVING GIVEN a first sketch of how synthetic technolo- 
gies can and will be used in practice, this chapter will provide 
definitions of the various technologies, describe the software 
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and hardware that enable synthetic AI, and explain the various vari- 
ants of the four technologies discussed in this book: Humanoid Robots 
(section 3.2), Deepfake technology (section 3.3), Augmented Reality 
(section 3.4) and Virtual Reality (section 3.5). The chapter closes with 
a short conclusion (section 3.6). After having given the practical and 
technical background, Chapter 4 will move to the heart of this book: the 
societal questions raised by Generative AI. 


3.2. HUMANOID ROBOTS 


Humanoid Robots (HRs) are essentially based on two fields of exper- 
tise: robotics and AI, where AI is often applied in combination with the 
Internet of Things. 

The Institute of Electrical and Electronics Engineers (IEEE) defines 
robots as agentive devices in a broad sense, intended to act in the physi- 
cal world to accomplish one or more tasks. In some cases, the actions 
of a robot might be subordinate to the actions of other agents, either 
software-driven or human. According to the IEEE’s definition, a robot 
is composed of suitable mechanical and electronic parts, through which 
it can form social groups, which may interact to achieve common goals. 
A robot or a group of robots can form a robotic system in special envi- 
ronments geared to facilitate its work. The International Organization 
for Standardization (ISO) defines robots in similar vein as ‘automatically 
controlled, reprogrammable multipurpose manipulator programmable 
in three or more axes, which can be either fixed in place or mobile for 
use in industrial automation applications’. It sees robots as suited to two 
main performative functions: 


e Tasks in personal use include handling or serving of items, trans- 
portation, physical support, providing guidance or information, 
grooming, cooking and food handling, and cleaning. 


e ‘Tasks in professional use include inspection, surveillance, handling 
of items, person transportation, providing guidance or information, 
cooking and food handling, and cleaning.* 


The term ‘Artificial Intelligence’ was first coined in the 1950s to describe 
a relatively new field of science, which was pursued by academia, the tech 
industry and the military to create human-like computer intelligence. In 
the 1970s, the first attempts were made to arrive at intelligent programs, 
such as in the field of speech recognition, with the US Defense Advanced 
Research Projects Agency (DARPA) putting in a particular amount of work. 
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These attempts were generally unsuccessful and, subsequently, invest- 
ments dwindled. It was only in the 1980s, largely due to investment from 
the Japanese government, that AI was reinvigorated, primarily in the field 
of so-called ‘expert systems’. In the 1990s, machine learning, statistical 
analytics and probabilistic computing were perfected, sparking a third 
wave of AI investments. Subsequently, the rise of ‘Big Data’ enabled the 
development of next-level intelligent robots, such as ‘robonauts’, smart 
drones and self-driving cars.” The Internet of Things powers the most 
advanced robots currently in use. 

Despite the famous Turing test,° the question of whether a robot is 
‘intelligent’ is not a binary one. Factors that play a role in assessing its 
intelligence include its capacity to learn from experience, to be flexible to 
changing environments and settings, to include all relevant sensory input 
in its decision-making process while ignoring irrelevant input, making 
appropriate estimations and decisions, and turning these into concrete 
actions that meet the goals being pursued. John Searle distinguished 
between what he called strong and weak AI.’ Weak AI refers to systems 
that are good at a relatively small or defined task, while strong AI is of 
at least comparable intelligence to humans. Most contemporary robots 
still fall into the weak AI category, but this is likely to change over time. 
Again, in reality, there is no binary distinction between weak and strong 
Al, but rather sliding scales and gradual differences. 


Figure 3.1 Representation of modern robots*® 
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Robots can be divided into several generations:? 


1 


Robotics 1.0: robots that have simple mechatronic structures with- 
out any degree of autonomy. 

Robotics 2.0: pre-programmed systems widely used in industry, such 
as for assembly lines and in nuclear facilities. 

Robotics 3.0: teleoperated systems that are used in space and 
medicine. 

Robotics 4.0: robots that are sensor-driven, allowing for intercon- 
nected and collaborative robots. Most current service robots that 
require human-robot interaction fall into this category. 

Robotics 5.0: robots equipped with autonomous capacities, allow- 
ing for more complex behavioural patterns and safety-critical tasks 
in the proximity of humans. Most self-driving cars belong to this 
category, as do many advanced surgical robots. 

Robotics 6.0: the newest generation of robots will integrate all the 
qualities of the previous generations and have more sensory input, 
greater cognitive abilities and a connection to the internet, enabling 
more advanced self-learning systems. 


There are multiple types of robots that can be distinguished based on 
their functionality, such as the following: 


1 


Swarm robots: robots that are usually less advanced and are coordi- 
nated using a central system. 

Co-robots: robots designed to aid humans in performing concrete 
tasks, such as industrial manufacturing work. 

Networked robots: robots connected to each other via a network 
and that coordinate and cooperate, usually to carry out a shared 
task. 

Cloud robots: robots connected to the cloud that depend on 
distributed computing facilities. They are able to share experi- 
ences and knowledge through the cloud, enhancing their learning 
capacities. 

Fog robots: robots that efficiently distribute computation and 
memory between edge, gateway and cloud; the ‘fog’ is usually closer 
to the end-user, bringing cloud capabilities down to the ground and 
making them generally more reliable. 

Drew robots: the tasks of these robots are distributed over a large 
number of devices, which are heterogeneous, ad hoc programmable 
and self-adaptive, enabling highly distributed applications that do 
not use central nodes. 
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Cognitive robots: robots that can behave intelligently thanks to 
their processing architecture, which allows them to learn and reason 
about how to behave in response to complex goals in a complex 
world. 

Smart robots: robots equipped with an AI system that can learn 
from its environment and experience and build on its capabilities 
using that knowledge. This makes their self-learning and directional 
capacity greater than that of cognitive robots. 

Ubiquitous robots: smart robots with additional technologies from 
ubiquitous and pervasive computing, sensor networks and ambient 
intelligence built into their systems. 

The Internet of Robotic Things: robots that connect with and use 
sensor and data analytics technologies from the Internet of Things 
to enhance their situational awareness, enabling more precise task 
execution. 


Figure 3.2 UN prize-winning humanoid robo 


119 


Source: UN Photo/Manuel Elias. 


The capabilities of HRs have increased dramatically thanks to rapid 
developments in sensory input hardware, technologies for process- 
ing input signals and increased ability to translate analytical insights 
directly into movement, speech and other actions. The distinction 
between non-humanoid and humanoid robots is not binary, but a 
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gradual spectrum.!! It depends on a variety of factors, such as their input, 
throughput and output capacity, and their appearance. Importantly, 
unlike other robots, humans are the benchmark for HRs. Thus, while 
most robots are instrumentalised to perform tasks in a better, quicker 
or more efficient way than humans (eg, because they can function more 
precisely and consistently, have more sensory input capacities or have 
better analytical skills), HRs proper should ideally have no fewer and 
no more capacities than a human. To be more precise, they should 
ideally stay within typical human limits. Perhaps the terms ‘Humanoid 
Robot’ and ‘Artificial Intelligence’ reflect the initial ideal of creating a 
human-like entity, while currently, the goal is to work towards a supra- 
human robot and Superior Intelligence. 

Equipping robots with AI generally means that they are better 
equipped to perform complex tasks and take appropriate decisions. 
Long-term autonomy (LTA), or the capacity to perform a series of 
complex tasks over time, is also increased by AI. LTA has been increased 


Humanoid Throughput 


Robots capacity 


Figure 3.3 The four factors that determine the ‘humanoidness’ of robots, with 
human capacities being the benchmark 
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by the rapid evolution in battery longevity and the rise of solar energy 
as well. Combined, these developments have meant that robots have a 
larger operational range. Owing to the widespread availability of the 
internet, robots can roam the streets in large parts of the world, getting 
process feedback through server or cloud connectivity.!* HRs generally 
use the ‘sense, plan, act’? model, comparable to the ‘input, throughput, 
output’ model used in computer sciences." 

To operate correctly, HRs include a wide range of technologically 
advanced hardware, such as cameras, microphones, speakers, motors 
and tactile sensors. These may provide information that is unavailable 
to humans, such as infrared or auditory signals that humans cannot 
detect. These input cues can be enhanced by information found on the 
internet, ranging from weather data to information about concrete 
topics or objects. AI robots can have a range of built-in software appli- 
cations, such as object recognition, computer vision software, facial 
and emotion recognition, natural language processing, Simultaneous 
Localisation and Mapping (SLAM), path planning and other high- 
level management features. In order to move and balance, robots 
are also equipped with magnets or alternative ways of directing their 
movements. 

The field of soft robotics is important in terms of appearance, which 
focuses on the use of materials and design: biomimetics:' 


Much of the research in soft robotics is linked to biomimetics, thus living 
organisms are often considered as a model, because they are masters to 
put in place simplifying principles that make their sensory-motor behav- 
ior efficient and highly flexible to respond to dynamic changes. The secret 
of natural systems lies in the smart characteristics of how their body is 
designed and in how their intelligence is embodied and distributed in it, 
allowing them to effectively adapt, grow and survive. And here lays the 
link with soft robotics: similarly, soft robots benefit of the use of smart 
and multi-functional materials (gels, elastomers, biological materials, etc) 
and of a body compliant with the external environment. Intelligence is also 
integrated in the body (‘embodied’) and co-develops with it, emerging from 
the interaction of the body itself with the world. This way, soft robots’ 
sensory-motor behavior becomes more efficient in responding to dynamic 
changes, as for living organisms. 


Although still in its relative infancy, the use of softer and more natu- 
ral materials is vital for creating a fully humanoid robot. In addition to 
the advances in the ‘naturalness’ of their outer appearance, the intui- 
tiveness with which they walk and talk in robot—human interactions has 
increased dramatically over the last decade. Soft robotics also makes 
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HRs more intelligent, because the material is equipped with sensors or 
responsive feedback systems. There are no perfect humanoid robots yet, 
but there are robots that are close to humanness on particular fronts, 
such as through their appearances or conversational capacities. 


3.3. DEEPFAKES 


Figure 3.4 Altering faces through DF technology!® 
Source: iStock/Getty Images Plus, FotografieLink. 


Deepfake (DF) is the name given to Machine Learning technology 
that uses AI and Deep Learning. Using Artificial Neural Networks 
(ANNs), which are based on and inspired by biological neural 
networks, systems learn how to perform tasks by looking at exam- 
ples, without specific rules being programmed into their systems." 
By discovering patterns, these networks can produce new sounds and 
images that are based on but are different from existing material. The 
most advanced technology for generating Deepfakes is Generative 
Adversarial Networks (GANs).?° Although the underlying technology 
is much older, the term ‘Deepfake’ was first used in 2017, a combina- 
tion of ‘Deep Learning’ and ‘Fake’. 

GANSs consist of two competing networks: a generator G(x) and a 
discriminator D(x).?! The two competing networks play an adversarial 
game in which the generator tries to ‘fool’ the discriminator by generat- 
ing data comparable to, but different from, the material in the existing 
training set; meanwhile, the discriminator tries not to be ‘fooled’ by 
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identifying fabrications.” The generator creates images from random 
noise (z) and learns how to generate realistic images. The input noise is 
sampled using uniform or normal distribution. After this, it is fed back 
into the generator, from which a generated image will emerge. The fake 
and authentic images are then fed into the discriminator, which learns 
how to distinguish them.” 


Real data a or 
Discriminator ————> Real/Fake 
G(z) 
Noise — i 
——> Generator 

(z) A i 
l | 
l | 

A ' 
1 | Backpropagation l 
Lans ee P as aa a a a as an es | 


Figure 3.5 Simplified representation of GAN? 


Using this technique by looking at, say, 1,000 photos of Donald Trump, a 
new photo can be produced. Although this technique is most frequently 
deployed to create pictures and videos, the technology can also be used 
for audio manipulation or to generate text or satellite data.” The tech- 
nique was initially complex, involved significant costs and required 
specialised equipment and programmes; now, it is possible for anyone to 
create a DF simply by using a smartphone.”° DF videos of a person can 
be created on the basis of a single photo,” and the technology is increas- 
ingly being deployed to bring to life high-resolution, whole-body images 
of ancient figures and non-existent people.?8 

It is best to conceive of the definition of DF as an ideal concept. Most 
DFs are videos created using advanced technological means in which an 
existing person appears to be doing or saying something that they never 
did or said, but it is impossible for the watcher of the video to discern 
this manipulation. Numerous peripheral applications to this core or 
ideal version exist. These include videos produced by low-quality techno- 
logical means (‘cheapfakes’), high-quality videos of non-existent people 
that appear to be real, fake audio clips? or manipulated satellite data 
and highly realistic minor manipulations of relative unimportance, such 
as smoothing skin tones. Six relevant factors are important for the ques- 
tion of whether material should be considered a DF: 


1) The type of data carrier and the extent to which the fake ‘infiltrates’ 
the consumer’s mind in its unfiltered form. For example, there is less 
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cognitive and emotional distance between a video and the consum- 
er’s perception than when it comes to a written text. 

2) The advanced or cutting-edge nature of the technology used for 
creating the material — generally, the more advanced this is, the more 
difficult it is to spot the manipulation. 

3) The degree of manipulation — the bigger the manipulation, the 
more the perception of reality is distorted. 

4) The degree to which the manipulation is essential or relevant to 
the context. Even minor manipulations can be essential in specific 
contexts; for example, equalised skin tones can produce inaccurate 
outcomes in a digital consult with the dermatologist. 

5) Whether the DF involves a living, deceased or non-existent person. 
The extent to which a DF of this variety has an impact on the 
consumer’s worldview may be similar for each of these, but with 
DFs of living persons, the real-world consequences are usually more 
substantial. 

6) The extent to which the user believes the material to be true. This 
perception is ultimately subjective, but it also depends on whether 
the consumer could see the person performing or saying the thing 
they are depicted as doing or saying. For example, hearing Barack 
Obama say ‘I’m the greatest human being on earth’ is intuitively 
less convincing than hearing another former US president saying it, 
although neither may have said it. 


Al-driven detection programs initially operated in a fairly straightforward 
manner, for example, by focusing on whether the person depicted in a 
video blinked, which was often not the case for inauthentic video, when 
DF technology first emerged.*! DF technology has since addressed that 
glitch, so detection programs now focus on spotting artefacts of manipu- 
lation or the absence of data commonly found in authentic media, that 
is, artefacts of normal/authentic media production.** DFs generated 
using GAN models are generally difficult to detect, as new output can be 
produced from existing input data, meaning that it cannot be compared 
with existing material and does not contain residuals from it.” 

In 2019, Facebook, Microsoft, the Partnership on AI coalition and 
academics from seven universities collaborated in the Deepfake Detection 
Challenge.** The best model was initially said to have an accuracy of 
82.56 per cent, but, after being tested again using a black box data set, 
the accuracy dropped to 65.18 per cent.*° Microsoft developed a Video 
Authenticator Tool in 2020, which can attribute an authenticity score to a 
photo or a video related to the probability that it is artificial.” There are 
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other technologies as well that can both detect manipulated content and 
verify authentic images.” Experts predict that, in time, the effectiveness 
of AI detection programs will decrease and an estimated 50 per cent 
success ratio will be reached. 


3.4. AUGMENTED REALITY 


The term ‘Augmented Reality’ (AR) is said to have been coined by Boeing 
researcher Tom Caudell in 1990, but the underlying technology is much 
older. As early as the 1950s, Morton Heilig worked on what he called the 
‘Cinema of the Future’, which allowed for interactive video screenings, 
and on what was known as the Sensorama, a machine that allowed for 
a multi-sensory experience of a film.” The machine can be considered 
as a prototype of Virtual Reality (VR), but is often also taken to be the 
start of AR. Ivan Sutherland subsequently invented the head-mounted 
display in 1966 and an AR system using an optical see-through head- 
mounted display two years later. Subsequently, rooms were designed that 
allowed users to interact with virtual objects.” By the 1990s, a number of 
major companies started using AR, inter alia, for visualisation and train- 
ing. Since 2008, the technologies that make AR possible have become 


Figure 3.6 AR based on see-through goggles 


Source: Getty — Westendé61. 
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powerful and compact enough to deliver AR experiences to personal 
computers and mobile devices.” 

AR allows for real-time direct and indirect viewing of the physical 
world in an enhanced or distorted way, by adding computer generated 
information to it.*! AR is often compared to VR, and it is sometimes 
described as the less scary and more promising variant of the two. AR 
devices like the Microsoft HoloLens and Magic Leap One superimpose a 
layer of digital content onto the real or physical environment: 


Typically, the digital objects — which can be anything ranging from a simple 
shape to a realistic model of a person — are rendered in stereo (i.e. with 
separate images projected into each eye) to give the illusion of depth when 
situated next to real objects. Moreover, the digital objects are registered, 
using cameras and sensors that track a user’s position in an absolute loca- 
tion, such that when a person moves, the object stays in the programmed 
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Figure 3.7 Milgram’s reality-virtuality continuum 


In the 1990s, Milgram proposed a virtuality continuum in order to 
stress the difference between AR and VR.* Others have suggested 
that AR should be seen as an alternative to both VR and Telepresence 
(eg, video calling a colleague from abroad in a meeting). While in VR, 
computer-generated 3D environments allow users to enter and interact 
with synthetic environments; in telepresence, the goal is to extend the 
user’s sensory-motor facilities to a remote environment: 


In this sense, telepresence can be defined as a human/machine system in which 
the human operator receives sufficient information about the teleoperator and 
the task environment, displayed in a sufficiently natural way, that the opera- 
tor feels physically present at the remote site. Very similar to virtual reality, in 
which we aim to achieve the illusion of presence within a computer simulation, 
telepresence aims to achieve the illusion of presence at a remote location. AR 
can be considered a technology between VR and telepresence. While in VR the 
environment is completely synthetic and in telepresence it is completely real, in 
AR the user sees the real world augmented with virtual objects. 
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AR can be roughly divided into marker-based and markerless applica- 
tions. Marker-based AR entails a camera that perceives a specific cue for 
the software to call up the correct information. Comparatively, markerless 
applications use positional data (eg, a mobile’s GPS or image recognition); 
input to the camera is compared to a library of images or other database 
to find a match. The latter version has wider applicability since it can 
function anywhere without the need for special labelling or supplemental 
reference points.“ Most AR systems operate through three steps: recog- 
nising, tracking and mixing. In the recognition phase, an image, sound 
or other input data is identified. AR systems generally require extensive 
calibration.*® During tracking, real-time location data and data about the 
object are analysed to adjust the superimposed layer to the perspective of 
the user. Finally, reality is mixed with synthetic media. 
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Figure 3.8 Different realities on the axes of authenticity and spatial 
convergence? 


Mobile devices have sparked the latest advances in AR:*° 


1) Cloud-based architecture: these systems operate in a client-server 
model with Mobile Augmented Reality Devices (MARs) executing 
the terminal applications, while the AR server is located in the cloud 
and is accessible through the internet. The server performs the most 
computationally intensive tasks. Its main advantage is that it allows 
for high-quality AR, while the mobile device itself can be light and 
use relatively little computational power. One disadvantage is that 
there is the delay and the required availability of internet access. 
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Edge-based architecture: these also operate in a client-server model, 
but the AR server is hosted at the network edge. This is more suit- 
able to serve latency critical applications, as edge servers at the core 
network provide a large number of services. One potential down- 
side of this mobile AR variant is that of a single point of failure, as 
failure of an edge AR server can potentially affect the entire system. 
Localised architecture: this functions with an AR server very close 
to the MAR device, or without the AR server, in which case the 
MAR device performs all the computing tasks. The advantage of 
this approach is that there is less or no dependence on the internet 
and a server, and the fact that the data stay on the MAR. One disad- 
vantage is the lower quality and shorter battery life of the MAR. 

Hybrid architecture: this integrates the three previous approaches, 
allowing AR to switch between or combine several platforms when- 
ever this is profitable in terms of functionality, efficiency and longevity. 
Although this approach eliminates many of the specific disadvantages 
of the other approaches, its disadvantage is that the combination of and 
switch between different types of architecture is not always smooth. 


AR tracking generally happens in one of the following ways:*! 


1) 


4) 


Fiducial marker-based tracking: in fiducial marker-based tracking, 
black-and-white squares with geometric figures are used as typical 
markers. This gives high contrast compared to the background envi- 
ronment, allowing for easy recognition. One of the disadvantages of 
this approach is that the objects always have to be visible and cannot 
be obscured during augmentation. 

Hybrid-based tracking: two or more data sources (eg, GPS, compass 
and accelerometer) are used to calculate position and orientation. 
This yields information about what should be augmented in the field 
of view without any actual processing of the real image (although it 
is used for placing the augmented layer). 

Model-based tracking: by manipulating their position, colour or 
orientation, representations can accurately resemble pre-determined 
3D objects. 

Natural feature tracking: objects in the real world can be used as mark- 
ers, focusing on peculiar features that distinguish them from others. 


Many early mobile AR systems primarily relied on mobile trackballs, 
trackpads and gyroscopic mice to support continuous 2D pointing 
tasks. Devices like computer mice are tangible and unidirectional; they 
communicate from the user to the AR system only. In most current AR 
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applications, these devices have been replaced by 3D equivalents, like 
paddles and wands.” These are called Tangible User Interfaces (TUIs) or 
bidirectional TUIs. Most of these are haptic User Interfaces (UIs). Haptic 
devices rely on both kinaesthetic (force, motion) and tactile sense (tact, 
touch). Instead of using hand-worn trackers, hand movement can also 
be tracked visually, such as through cameras pointed at the user’s hands. 
Gaze tracking has recently attracted considerable attention, as have aural 
solutions and speech commands. Biometric devices can measure heart 
rate and bioelectric signals, such as galvanic skin response, neural activ- 
ity and muscle activity; these signals are used to feed biological activity 
back into the AR. 

AR devices work through various display types, such as the 
following:* 


1) Monitor-based: the least futuristic, but currently the most used 
form of AR is where the image is shown on a desktop monitor or a 
mobile device (eg, Google Lens). 

2) Optical see-through: these systems are based on a transparent, 
head-mounted display to show the virtual environment directly 
over the real world. It is placed in front of the user’s eyes; the mate- 
rial of the display can be glass, plastic or any other transparent 
material. 

3) Video see-through: a see-through device is typically placed on the 
head. However, it does not project a layer on the glass through 
which reality is seen, as in the second approach. Instead, it makes 
a combined projection by filming the environment and projecting 
those shots, with the additional artificial layer, on the screen in front 
of the user’s eyes. 

4) Virtual retinal: this version of AR is based on the projection of 
Al-generated content directly onto the retina of the eyes of the 
users, producing a rasterised image. 


3.5. VIRTUAL REALITY 


Arguably, it was Jaron Lanier who coined the term ‘Virtual Reality’ in 
1989, giving rise to the first wave of VR technologies and applications. 
Many of these were used in the entertainment sector. It was only in 2012, 
with the introduction of Oculus Rift, that a second and more advanced 
wave of VR products reached the market. 
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Figure 3.9 VR used in school 


Source: Getty — izusek. 


A distinction is often made between three types of VR:°° 


1) Desktop VR: this uses subjective immersion, which is less realistic, 
although the feeling of immersion can be enhanced through stereo- 
scopic vision. 

2) Cave VR: this operates in a small room specially designed for VR 
experiences in which a computer-generated environment is projected 
onto the walls.** Most Cave VR systems work in a two-stream fash- 
ion, depending on both physical to virtual synchronisation and also 
virtual to physical synchronisation.” 

3) Fully Immersive VR: the user finds themself in a fully Al-generated 
environment and gets sensory input from that environment through 
visual and audial display and force feedback from sensors. 


As with AR, there are various forms of computing that can assist in 
designing VR, such as Mobile Cloud Computing, Edge Computing and 
Fog Computing.” VR hardware can be divided into input and output 
hardware.’ 

In terms of the input devices, several categories can be distinguished: 


1) Controllers: commonly hand-worn and allow for input from buttons 
or touchpads, for example, one joystick for each hand. These input 
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devices may either be wired or wireless, or, if multiple input devices 
are used, a combination of both. 

2) Navigation devices: the user can be provided with the illusion of 
moving through space while staying relatively static through the use 
of navigation devices. 

3) Body tracking devices: gestures and other body signals can be 
tracked by body tracking devices, such as through magnetic data 
gloves. 


Four types of output devices can be distinguished: 


1) Mobile Head-Mounted Displays (HMDs): these are usually 
deployed wirelessly and are able to operate without an additional 
computer. Such devices could be add-on frames for a smartphone, 
an ergonomically designed smartphone case or a stand-alone system 
specifically designed for this purpose. 

2) Wired HMDs: most stationary HMDs are connected to a powerful 
computer, but they still allow the user to move around the space, 
usually a secluded room. This type is particularly suitable for both 
gaming and training purposes. 

3) Haptic devices: these are generally worn on the body, but they can 
also be carried by the user. 

4) Multi-sensor devices: these allow for a combination of audio, video, 
haptic and/or olfactory sensory output. 


The latter category already shows that although most VR systems focus 
on producing visually realistic information, other senses may be equally 
or even more important.” Aural cues can either be integrated into the 
environment by having speakers in the room (eg, in the cave or behind 
the desktop) or more directly, through in-ear or on-ear devices. Haptic 
sensory signals have been used for a long time, if only the rumble pack 
integrated in joysticks or controllers since the 1990s, which give the user 
a vibratory feeling when, for example, their avatar hits a wall or gets 
punched. Smell is perhaps still the least-developed sense in Al-simulated 
worlds, although there have been significant developments in this field: 


Virtual olfaction is ... defined as the act of smelling an odorant produced 
by a virtual olfactory display. The last concept is teleolfaction, defined as 
the act of smelling a mixture of odorants whose composition is related to 
a mixture present in a remote place. Odorant storage is, perhaps, the most 
mature of the various technologies required for a virtual olfactory display. 
Odorants can be stored in several ways, including liquids, gels, or waxy 
solids. The most popular storage method for previous and current VE-related 
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work seems to be to microencapsulate odorants: droplets of liquid are encap- 
sulated in a wall of gelatin and printed on a flat surface using silk screen 
techniques. This method is the basis of scratch-and-sniff patches, where the 
odorant is released by subjecting the particle to mechanical shear or melting 
the gelatin wall. Microencapsulation offers the advantages of discrete meter- 
ing of odorant dosage, stability at room temperatures, and the unlikelihood 
of messy spills. Other commonly used methods include air dilution olfac- 
tometry, breathable membranes coated with a liquid odor, and a system of 
liquid injection into an electrostatic field with air flow control.®! 


It is still not exactly clear how identification with avatars in VR works, 
although many experiments have established that our brains tend to 
identify with extended versions of ourselves. Drivers are familiar with 
how quickly cars start to feel like extensions of the body. Even in the 
classic experiment, where a participant has to place their hand under the 
table and a rubber hand laid on the table where their hand could have 
been, and a hammer is smashed on the rubber hand, the brain and body 
reacts, even though the participant is consciously aware that this is not 
their hand. Because the VR world appears to be real, the user sees the 
world through the avatar and receives all kinds of stimuli that they see 
as corresponding with what the avatar is experiencing; consequently, a 
strong identification is generally established. Even although they cogni- 
tively know that the avatar is not them, they emotionally identify with it, 
sometimes even more so than they do with their ‘physical avatar’, if only 
because the appearance of the virtual avatar is customised to the player’s 
preferred body representation.® 


3.6. CONCLUSION 


This chapter provided a brief sketch of four Al-driven technologies that 
affect our reality and our perception of it. Each of these technologies 
does so in distinct ways. 

Ideally, HRs are indistinguishable from human agents. However, 
most contemporary HRs only excel in one or two domains, such as their 
physical appearance, their social skills (eg, an advanced chatbot) or their 
ability to perform one specific task (eg, cleaning, sex or manufacturing), 
but holistic robots are in the making. Some HRs are designed in a way 
to resemble a specific person, such as a deceased loved one, but most 
have their own ‘character’. With the more advanced robots, their char- 
acter will evolve and change over time, through integrated self-learning 
systems based on engagement with humans and the environment. 
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DFs aim to be indistinguishable from authentic material, yet do so 
on a representational level. When deployed for satire, DFs are often so 
remarkable (eg, Trump doing a tap dance) that the consumer knows it to 
be fake, but the cues are found in contextual information (ie, this is not 
something this person would do) rather than from technical limitations — 
otherwise, DFs are often indistinguishable from authentic material. 
There are fully virtual DF people, which, like humanoid robots, do not 
represent a particular existing person or a person that has existed, but 
appear to be unique beings.“ 

With AR, the perception of the consumer that the Synthetic Reality 
is Physical Reality (PR) or an accurate representation of it is not always 
important. It may be that when olfactory reality is altered, this subcon- 
sciously influences an oblivious citizen, and when combatants only train 
in augmented environments where the sight of blood is obfuscated, they 
may, in time, forget about the Al-affected nature of their reality. However, 
in most current forms of AR, such as when smiley faces are placed on the 
right track, when information is projected next to historical objects or 
when a virtual assistant gives an audio tour of the Egyptian pyramids, 
the consumer knows and remains aware of the augmented nature of 
their reality. Gamification is highly dependent on the artificiality of the 
perceived reality: examples are Disney-esque sounds and visual cues that 
turn ordinary tasks into fun experiences. 

In VR, Physical Reality is replaced in full. VR may but does not neces- 
sarily create a realistic world or accurate representation of the world; 
especially in gaming, it allows the user to explore a new, virtual world. 
Likewise, a user’s avatars may or may not resemble the user. Although 
identification is typically more easily established in an environment that 
resembles the world as the user knows it, this is not necessarily true for the 
avatar. Identification also depends on the ideal body projection. Matrix- 
like scenarios, in which people live in VR from cradle to grave, are still 
a long way off, so contemporary and near-future VR will entail users 
consciously entering VR, although they may forget about its virtuality 
when they stay in that reality for longer periods of time. An exception 
may be young children or those that have cognitive limitations, such as 
people suffering from Alzheimer’s disease. 

All four technologies are highly dependent on visual exactness. For 
HRs, this concerns the physical appearance of the robot; for the other 
three technologies, this entails the representational exactitude. Even with 
virtual creatures in VR, their representation should be consistent and 
intuitive, and they should meet users’ expectations of how these crea- 
tures should look, move and speak. Audio is very important to HR and 
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DE, with the question of whether a natural person would speak like the 
HR and DF being the benchmark. Speech recognition, natural language 
processing and conversational software are important in this respect. 
Smell is sometimes used in AR and VR and olfactory experiences can 
now be fully Al-generated.© It could be used with HRs, but so far, this 
type of application has not taken off. In DFs, no expected olfactory 
application is foreseen. Haptic Al-generated input is important for AR 
and especially for VR, such as in gaming. For HRs, haptic stimuli are 
often integral to interaction as the HR is placed in Physical Reality. It is 
important that the feel and touch of a human are represented; advances 
in the field of soft robotics are being made. Gustatory input systems 
seem to be mostly absent for any of the four technologies, although they 
can be envisaged for HR and VR. Taste, like smell, sound and imagery, 
can be produced artificially. 

DFs and AR are generally unidirectional: DFs can be consumed as 
a final product and AR generally produces additional sensory input in 
the form of visual or audio cues. Still, AR can be interactional, such as 
if the user asks their virtual tour guide questions when walking around 
the pyramids. With DFs too, there are applications in which, for exam- 
ple, a person can make a video call with their face replaced with that 
of a politician. In these instances, DF technology translates the facial 
expressions, lip movements and speech acts of a real person to that of 
the impersonated person in real time. VR and HRs depend heavily on 
their interactional faculties. In VR, the avatar needs to be able to interact 
with the environment and with other (AI or human-driven) avatars. This 
generally holds true for HRs as well, which need to be able to respond to 
their environment, humans and other HRs. For HRs that have one task, 
such as manufacturing, this is less the case. 

There are questions as to delineation with each of the technolo- 
gies. For example, there is discussion as to whether and to what extent 
augmented sound and smell should be seen as AR. For example, play- 
ing music from a CD, smart device or other medium is generally not 
seen as a form of AR, but when sounds are played as corresponding with 
an artificially produced visual image, this might be the case. Is hearing 
a virtual tour guide when at the Pyramids of Giza essentially different 
from listening to the same tour guide while viewing computer images of 
the pyramids? How is it different from listening to a virtual tour guide 
when looking at photos of the pyramids? How is it different from listen- 
ing to the tour guide at the physical pyramids on a CD? There is a similar 
discussion with respect to perfume and artificial odours. Is spraying pine 
resin perfume in the house already AR, or is it only AR when looking at 
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grass onto which Al-generated pine trees are projected? Or is the deter- 
minative factor whether the stimuli themselves are created through AI — 
that is, should AI create the scent or is it enough if AI determines when 
and how the scent is sprayed? A human with a robot arm is clearly not a 
HR, although it is a way in which artificially generated reality is merged 
with non-Synthetic Reality. But what if more body parts are replaced 
and a person’s speech abilities are computer-aided? There is no clear-cut 
distinction between human and a HR, or between a HR and a non-HR. 

Finally, there is overlap between the various concepts, especially 
between DFs, AR, and VR, and many of the underlying technologies are 
shared by each of the four technologies. VR is not clearly distinguishable 
from video see-through AR, in which PR is copied and integrated with 
Al-generated content in a representational image, sound etc. Similarly, 
monitor AR and desktop VR are closely related, and they are not sharply 
distinguishable from interactive DFs, which allow users to interact with a 
fully virtual humanoid avatar in a video-call. Chatbots are often catego- 
rised as prototype one-dimensional robots, but they could also be seen 
as DFs, especially when the chatbot is granted an acoustic voice and a 
realistic outer appearance. In such scenarios, multiple technologies are 
applied simultaneously. 
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Figure 3.10 Categorisation of the four techniques on two axes (interactionality 
and reality) 


A 
Societal Challenges 


Al can be — and reportedly is — used to disrupt democracy through interfer- 
ence in electoral processes, personalised political targeting, shaping voters’ 
behaviour and manipulating public opinion. Furthermore, Al has seemingly 
been used to amplify the spread of misinformation, ‘echo chambers’, propa- 
ganda and hate speech, thus eroding critical thinking, contributing to rising 
populism and the polarisation of democratic societies. 


Moreover, the broad use by States and private actors of Al-based technologies to 
control individuals, such as the automated filtering of information amounting to 
censorship, mass surveillance using smartphones, the gathering of personal data 
and tracking one’s activity online and offline may lead to the erosion of citizens’ 
psychological integrity, civil rights and political freedoms and the emergence of 
digital authoritarianism — a new social order competing with democracy. 


The concentration of data, information, power and influence in the hands of 
a few major private companies involved in developing and providing Al-based 
technologies and services, and the growing dependence of individuals, insti- 
tutions and society as a whole on these services, are also a cause for concern. 
These big companies no longer serve as simple channels of communication 
between individuals and institutions but play an increasingly prominent role 
on their own, controlling and filtering information flows, exercising auto- 
mated censorship of content published on social media, setting the agenda 
and shaping and transforming social and political models. Acting on the 
basis of business models that prioritise the profits of shareholders over the 
common good, these actors may be a threat to democratic order and should 
be subject to democratic oversight. 


Parliamentary Assembly, Council of Europe 


4.1. INTRODUCTION 


HAPTER 2 ALREADY mapped some of the direct effects (both posi- 
tive and negative) of synthetic technologies. This chapter will 
home in on the long-term societal consequences and evaluate 
how values functioning as the pillars of contemporary society may be 
reconfigured in a synthesised world. Obviously, these long-term effects 
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are undetermined and unknown to a large extent. New effects may 
emerge that are currently inconceivable; effects that are now materialis- 
ing may be curbed or tackled, for example, through legislative changes. 
Predictions about the direction in which the technologies will evolve are 
not a matter of exact science, and this holds even more true for the soci- 
etal changes these technologies trigger. 

This means that this chapter, although grounded in established theo- 
ries and current understandings of the technologies and their effects, 
is more suppositive than the previous two. However, it is arguably also 
the most important one, as it deals with the foundations of and condi- 
tions for human interaction, individual autonomy and the democratic 
rule of law. It is important to think through how these core values may 
potentially be affected well before any changes materialise, as in order 
to prevent or curb problems, it is often necessary to act pre-emptively 
instead of retroactively. Once a technology has already been introduced, 
widely deployed and its effects have already materialised, it is generally 
almost impossible to turn back the clock. 

This chapter does not deal with all the potential societal effects the 
technologies may have, both because there are simply too many and 
because most of them are not directly linked to the core theme of this 
book: reality and our perception of it. Instead, four core aspects of human 
interaction and democratic societies are discussed in the subsequent four 
sections of this chapter. These are: the influence of Synthetic Reality on 
our perception of truth and trust in societal institutions (section 4.2), its 
impact on human relationships and social interaction (section 4.3), its 
effect on individual autonomy and societal equality (section 4.4) and, 
finally, questions it triggers over the role of technology in society and the 
attribution of rights to non-human entities (section 4.5). A conclusion is 
provided at the end of the chapter (section 4.6). 


4.2. TRUTH AND TRUST 


The technologies discussed in this book may have a substantial impact on 
reality and our perception of it. Some of them will be small, others big, 
some will materialise in the near future, while other effects will become 
apparent only in the long run, some will concern high impact incidents, 
and others involve minor but structural effects and, as such, will have a 
big cumulative impact on society. 

It is obvious that Synthetic Reality (SR) can have a direct impact on 
Physical Reality (PR). A Deepfake (DF) of a person in which they are 
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seen making racist jokes can have a significant impact on their reputa- 
tion. If an employee is convinced they are speaking to their boss and 
transfers money, this has real and tangible repercussions both for them 
and for their company. The same goes for false news stories about a 
Muslim desecrating a Hindu religious object, which may lead to hate 
crimes and violence, and suggestions about banks being on the verge of 
collapse, which may cause stock prices to plummet. Also, if people do 
not travel to polar regions but use Virtual Reality (VR) to do so instead, 
this has real and tangible effects on global warming. 

Even if people know that the information presented to them is false 
or Al-generated, the consequences can still be real. A person may know 
that an avatar resembling their friend is not controlled by that friend, yet 
seeing the avatar resembling their friend have sex with a virtual animal 
may still (unconsciously) affect their perception of that friend. A DF 
distributed at a high school of a girl having sex with multiple men, even 
if her classmates know that video to be fake, may nevertheless shape 
their view of her. Seeing herself perform these sexual activities may 
shape a girl’s self-image as well, even knowing that she never engaged in 
those activities. This also applies for people who are treated for chronic 
pain or dissociative disorders in VR; seeing their avatar act in the virtual 
world may still leave real and direct imprints on their mind and their self- 
image (both subconscious and conscious), even though they rationally 
understand the avatar is not them. 

Sometimes, people do not care about the veracity of specific content 
if they believe the underlying message to be true. It may well be the case 
that a group knows the girl from the example above did not perform the 
sexual activities, but if she is known to be a ‘slut’, the video, although 
untrue itself, may be seen as confirming an underlying truth. It may well 
be that Hillary Clinton never said that she thinks that those who voted for 
Donald Trump are lower than vermin, and a video in which she appears 
to be saying so may be so evidently false that it is clear to anyone that it 
is fake; all the same, Trump supporters may still share the video because, 
to them, it reaffirms their pre-established convictions about Clinton. 
The same goes for VR worlds that depict the state of the world in 30 
years’ time if world leaders continue to be reticent in addressing climate 
change; although it is clear to climate activists that it is a VR world, to 
them, this may be the reality that most policy makers are denying — it 
may be the ‘actual truth’, while policy makers continue to believe in a lie. 

When a falsehood is first spread and later debunked, the effects may 
be real and irreversible: money may already have been wrongfully trans- 
ferred, violence may already have been incited against minorities or a 


62 Societal Challenges 


political leader may already have dropped out of an election. In addition, 
as with seeing content a person knows to be Al-generated, news that is 
later debunked will leave a lasting impact. First, fake news is often more 
prominently distributed and reaches more people than the later arti- 
cles debunking it. Second, the falsehood is often more sensational and 
thus leaves a bigger imprint on a person’s memory than the later work 
debunking it. Third, it is known that denying a falsehood often leaves a 
hint of the falsehood in a person’s memory. A politician explicitly deny- 
ing that there are B-3 bombers at a press conference will lead people to 
think that there might be B-3 bombers.* Fourth, even if a person believes 
the debunk, over time, people tend to forget factual information, leaving 
a general ‘wasn’t there something with ...’ feeling. 

Amore structural impact of the increased production of Al-generated 
content is that people will be unsure what they can believe. Parents who 
are unsure whether it is really their child who is calling them in a panic 
because their wallet has been stolen may want to verify and double 
check before sending money. People who have mistakenly believed in a 
falsehood before are known to be more cautious when seeing shocking 
or sensational news. Insecurity about the veracity of communication in 
a world that is almost fully digitised and mediated can be significant, 
both on a personal and a societal level. The already declining trust in 
‘mainstream media’ may accelerate, and the trend that people choose 
the media that reaffirms their pre-established worldview may deepen. 
This may mean that groups become increasingly transfixed by their 
own perceptions of reality, which may lead to polarisation and soci- 
etal discontent, a trend that is already visible in the US, according to 
experts.’ They also point to the impact of echo chambers and epistemic 
bubbles on society.* 

These phenomena may mean that people will avoid the news, while 
a well-informed citizenry is a precondition for a vital democracy. In 
addition, because there is so much Al-generated material online and 
because this material is increasingly difficult to authenticate or distin- 
guish from non-Al-generated material, this may feed into a trend that 
is already ongoing: reality fatigue. People may become increasingly 
indifferent to reality and authenticity, as they know what is real and 
what is not may be very difficult to establish, and because so much of 
‘actual’ reality is already shaped or affected by AI. People may create 
virtual worlds for themselves and their peers and spend most of their 
time there, or create their own news bulletins using DF technology. 
They can design their worlds according to their own preferences and 
worldview. 
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Al-generated content can also have a consolidating effect on the 
perception of truth. AI is generally based on historical data, distils 
patterns from them and takes these data as predictive of the future. In 
addition, AI devices, such as ChatGPT and Augmented Reality (AR) 
devices, are likely to show and thus confirm the dominant version of 
the truth, such as when ChatGPT is asked about the causes of climate 
change or the fairness of American elections, or when an AR device 
projects information next to objects or people, instead of giving several 
accounts of the truth or showing dissenting opinions on controversial 
topics. An additional problem is that Large Language Models learn 
from content that is fed to them, but because content will be increas- 
ingly wholly or partially Al-generated, it will be partly learning from 
itself, thus creating a feedback loop. The opposite effect of Generative 
AI has also materialised, namely when AI fabulates and makes up 
entirely fabricated facts, news events and historical figures, presenting 
them as real and authentic. 

To complicate matters, Al-generated content often goes beyond our 
current, binary perception of the truth. Is a DF avatar influencer that 
has sponsor contracts and a following real or unreal? Is an Al-generated 
painting or concerto authentic or not? Is a VR interaction between two 
avatars real or fake? Is a homework assignment generated by ChatGPT 
truthful or not? Is being able to see ultraviolet through an AR device real 
or fake? These examples do not exactly lend themselves to being catego- 
rised as real or fake: they fall either somewhere in between or, perhaps 
more accurately, form new realities. 

Many forms of SR not only have a concrete effect on PR, but they 
take place in it too. Humanoid Robots (HRs) are physically present and 
operate in the ‘real’ world, AR tour guides operate at historical sites and 
DF technology is used for lip-syncing and live-translating conversations 
between two employees with different languages speaking over a video 
call. On a more philosophical level, if 95 per cent of the people see an 
avatar at a party and engage with it, is the avatar and the interaction with 
it less real than what the other 5 per cent of the people see? 

In addition, the fact that there are so many falsehoods around and 
that it is increasingly difficult to distinguish between AI and non-AI- 
generated material mean that a claim that a true event or a fact is or 
may be false will intuitively be more appealing. Thus, the consequence 
might not only be that untrue events are perceived to be real, but that real 
events might also be perceived to be unreal. This means that a person 
convicted for a crime can always maintain that they were convicted based 
on false evidence and that it may never be fully determined whether this 
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Figure 4.1 A non-exhaustive overview of the various ways in which PR and SR 
interact 


is correct or not. Already, according to some polls, around 70 per cent 
of Republicans in the US believe that Joe Biden did not legitimately win 
the 2020 presidential election. When synthetic technologies are used to 
change or challenge reality, the percentage of people who may believe 
real events to be unreal may even be higher. 

The four technologies described in this book fit the general trend that 
increasing amounts of digital content are manipulated by default. This 
often relates to relatively minor manipulations: video call services that 
equalise users’ skin tones by default, audio communication in which 
higher sound registers are lost or marginalised by default as an effect of 
compression. Even these smaller manipulations can be of great impor- 
tance in specific circumstances, such as when identifying a suspect or 
in an online medical consultation with a dermatologist. Experts predict 
that, in about four years’ time, more than 90 per cent of all digital content 
will be synthetic — that is, it is material that has been wholly or partly 
manipulated or generated by digital means.° 

Detection programs sometimes only give an ‘authenticity percentage’, 
for example saying ‘the chance that this video is authentic/not manipu- 
lated is 73 per cent’. Experts argue that the best strategy for detecting 
DFs is not through counter-technology, but through human appraisal 
of contextual information: in other words, is this something this person 
would normally say? Are there other sources that confirm the report? 
However, AI is also being used to create fake environments: not just DF 
video, but also fake news websites that report on it, fake X accounts 
that discuss the fake video, fake Instagram accounts that generate memes 
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using frames from the video, fake Wikipedia pages that are automatically 
updated or created on the subject matter, fake new items that report on 
the issue and so forth. Creating a fake environment makes it difficult 
for humans and algorithms to distinguish authentic from inauthentic 
material. 

These effects on truth also have huge repercussions for trust, as the 
two values are deeply interrelated. Trust is foundational for people’s 
perception of reality. People trust that the sun will rise the next morning, 
that their friend will still be their friend tomorrow and that they will still 
be able to walk next day. Although changes of PR do occur, they usually 
do so gradually. There may come a time when the sun will explode, but 
itis likely that for the next millennia, it will continue to rise in the morn- 
ing; friendships do dissolve, and sometimes abruptly, but most often they 
are stable or gradually peter out over time; although medical issues might 
materialise overnight, mostly, our physical and psychological selves 
remain relatively static and change only moderately over time. This epis- 
temological stability provides people with a basis for their perception of 
the world, others and themselves. If a person is unsure whether they are 
going to live or die the next day, this has huge repercussions on their abil- 
ity to engage with others and find meaning.’ 

Al-generated reality can be stable or even static, but it does not need 
to be, precisely because it is not grounded in PR and has no physical 
limitations. In VR, a person’s avatar may be stolen, the virtual world 
may be redesigned, the rules of a game may be changed, an avatar friend 
may appear to be controlled by a bully from work or perhaps controlled 
by a stranger who feels the need to live out an unexplored part of their 
personality and thus change their character and, depending on the plat- 
form, physical appearance overnight. HRs also do not need to be stable 
in terms of character. When equipped with self-learning software, they 
may develop in ways that were unforeseen by their programmers. These 
types of radical changes can have a significant impact on a person’s 
worldview and lead to existential fear, anxiety and insecurity. 

A synthetic society may cause epistemological insecurity: is what lam 
seeing actually true? Am I actually speaking to the person I think I am? Is 
this a human or a HR? Is the avatar resembling a friend really controlled 
by that friend? Is the avatar a ‘truer version’ of the natural person 
controlling it or is it a deceptive version of them, or are the physical and 
the virtual avatar both equally important representation of one person? 
Is the avatar human or Al-generated and/or controlled?! Combined 
with a form of hyperpersonalisation, where everyone lives in their own 
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reality and the ‘sequestration of experience’, it may be difficult to verify 
whether what one has experienced is true: ‘Hyperpersonalisation can 
thus lead to a loss of confidence in (objective) perception. What value 
can we still assign to a story or an eyewitness account if we cannot judge 
whether what the person has actually experienced is ‘real ? 1 

Another opposite effect, which can exist simultaneously, is that SR may 
become almost fully static and represent a nostalgic dream world where 
everything is as a person wants it to be. Because, in PR, most things grad- 
ually change and some things abruptly, a person is continuously forced 
to adapt to new circumstances and to interact with new and unforeseen 
events. In doing so, they develop new skills and their identity evolves. When 
a person continues to communicate with a deceased partner through DF 
technology, lives in the perfect VR world with their avatar representing 
their idealised self-image or when a humanoid love robot fulfils a person’s 
every desire precisely as programmed, people do not have to face the real- 
ity that their spouse has passed away, that they are morbidly obese or that 
they cannot find a partner, which may lead to people developing static and 
one-dimensional characters. Living in a synthetic reality partially removes 
the need for trusting others in interpersonal relationships. 

Finally, at least three societal institutions are affected by SR. 

First, the rise of SR will mean legal procedures will take substantially 
longer because parties can claim that evidence provided against them is 
fabricated. This might require verification procedures and lead to discus- 
sions in court between various experts who each have their own views 
on the likely authenticity of the material. In addition, when 90 per cent 
of all digital content is wholly or partially Al-generated or manipu- 
lated, there may come a point at which every piece of evidence needs to 
be verified on authenticity as a standard (see section 5.5 in Chapter 5). 
This means considerable delays, added costs and increased bureaucracy. 
There is also a risk that a judge will wrongly assume content to be true, 
leading to unjust decisions. The reverse, namely that the judge believes 
that certain material is (possibly) fake when it is not, and thus excluding 
it, can also have considerable undesirable consequences. Moreover, for 
certain offences, the mere suggestion of criminal conduct can be enough 
to incite public outrage, even if it is later established that content was 
fake. The growing trend of trial by media may thus worsen. 

Second, the rise of synthetic media will have an impact on the func- 
tioning of the press. The media is already struggling to properly check all 
User-Generated Content and online material for accuracy and authen- 
ticity. A small but telling example is a football player who was carried 
off the pitch during a European Championship match because of a 
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medical condition, shortly after which a picture appeared on Twitter 
(now known as X) that would prove that he was still alive when he was 
taken to the hospital. However, it took quite a while for many traditional 
media to mention the photo, as it might have been a fabrication or a 
postdated image. In a world where every citizen has access to synthetic 
technologies and can create and distribute fake videos, photos or audio 
files within minutes, the question is how media can ensure in practice 
that their reporting remains accurate. Quality media that invests in such 
procedures not only runs the risk of making less profit because of the 
cost involved, but also of becoming ‘obsolete’, because other media, with 
less due diligence, would be quicker with coverage and post sensational 
stories, even those that would later turn out to be false. 

Third, SR can affect the democratic process. Foreign powers, especially 
Russia, are already known to deploy fake news and trolls to influence 
democratic elections. It is also clear that special interest groups are using 
synthetic media within national borders in ways that favour their political 
allies and disadvantage their political rivals. In addition, experts point out 
that states try to influence concrete decision making relevant to them in 
other countries, such as in the Global South. Synthetic media can have a 
potentially highly disruptive effect on the political process there as well as 
on the international legal order, as DFs may be used to affect governments 
or governmental votes at the level of the United Nations, for example, 
when voting on sanctions to be imposed on Russia. In more general terms, 
in Western societies, trust in democratically elected leaders is low and, 
in most countries, voter turnout is declining. The rise of fake news and 
SR may further deepen this trend, as well as undermining one of the pre- 
conditions for a vital democracy: a well-informed citizenry. 


4.3. HUMAN RECOGNITION AND SOCIAL INTERACTION 


The German philosopher Georg Wilhelm Friedrich Hegel wrote a para- 
ble about the first time two humans encountered each other. They see 
themselves reflected in each other, which deprives them of their previ- 
ously presumed unique position: that of being the only human on Earth. 
Interestingly, for Hegel, it is only at this moment that both become 
‘proper’ humans. Hegel suggests that in order to become human, we 
need to be recognised as a human by another human. It requires person 
A to see person B as a human and person B to see person A as a human, 
as well as person A recognising that person B sees them as a human and 
person B recognising that person A sees them as a human. Only then 
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do A and B become fully human. For Hegel, seeing another person as 
a human ultimately means that they are not an object, but a subject, 
and not a means to an end, but an end in themselves. This means that 
being human by its very nature entails being relational, that individual- 
ity can only exist through mutuality and that treating another person 
as a human entails the opposite of instrumentalising the other for one’s 
own needs. Mutual recognition means that the ‘other’ is no longer seen 
as a stranger, but is recognised as part of the ‘self’, and that the ‘self’ is 
reflected in the ‘other’." 

But Hegel also gives a darker account of what happens next in this 
fictional encounter. He stresses that both A and B want to retain their 
uniqueness and individuality, and so try to annihilate each other. In the 
subsequent struggle for life and death, one wins. A becomes the lord and 
B the bondsman; A is a subject and B is the object; A is the end and B is 
the means to that end. Hegel suggests that neither one consequently is 
human, because the recognition process is no longer between equals and 
that although it seems that A is in the best position, in a twist of fate, it is 
B that over time becomes fully human rather than A, as will be explained 
further down below. 

This parable is important for the technologies discussed in this book 
for several reasons. To start with, the question is how a process of mutual 
recognition takes shape when there are more and more interspecies or 
human-robot/avatar relationships. There is already abundant evidence 
that shows that, at the very least, digitised social interactions are incom- 
parable to social interactions in PR. For example, Sherry Turkle, a 
leading scholar who embraced social technology at first, seeing it as a 
fruitful way to establish new forms of social engagements, subsequently 
became sceptical: 


Online, we easily find ‘company’ but are exhausted by the pressures of 
performance. We enjoy continual connection but rarely have each other’s 
full attention. We can have instant audiences but flatten out what we say 
to each other in new reductive genres of abbreviation. We like it that the 
Web ‘knows’ us, but bread crumbs that can be easily exploited, both politi- 
cally and commercially. We have many new encounters but may come to 
experience them as tentative, to be put ‘on hold’ if better ones come along. 
Indeed, new encounters need not be better to get our attention. We are wired 
to respond positively to their simply being new. We can work from home, 
but our work bleeds into our private lives until we can barely discern the 
boundaries between them. We like being able to reach each other almost 
instantaneously but have to hide our phones to force ourselves to take a quiet 
moment ... We brag about how many we have ‘friended’ on Facebook, yet 
Americans say they have fewer friends than before. When asked in whom 
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they can confide and to whom they turn in an emergency, more and more say 
that their only resource is their family. The ties we form through the Internet 
are not, in the end, the ties that bind. But they are the ties that preoccupy.!* 


Social scientists suggest that the substitution of online contact for physi- 
cal contact could have a significant impact, as people’s sense of belonging 
declines, leading to more stress and anxiety.” In addition, it is known that 
it is more difficult to recognise emotions in online interactions, which is 
even more true when DFs, AR or VR are used. This has concrete, practi- 
cal and long-term effects: people will become increasingly less capable 
of recognising emotions in others, and this will carry over to PR, simply 
because they are no longer conditioned to do so. 

Consequently, virtual friendships, either with the avatar of a physi- 
cal person or a fully Al-generated and controlled avatar, may, in the end, 
increase feelings of loneliness and reduce a person’s sense of belonging. 
With HRs, it could be argued that they do not suffer from this disadvan- 
tage as they engage with humans in PR. If the dream of a truly humanoid 
robot would come to fruition in time, meaning that robots would look 
like humans physically, intellectually and emotionally, this might give rise 
to new and valuable social interactions and allow for new forms of inter- 
species recognition and interdependencies. An experiment with a not 
so advanced robot named Alice entailed her being placed with elderly 
people living alone in a retirement home. Although she did not look like 
a proper human, and her social and intellectual capacities were rather 
limited, when she was taken away after a month, most participants 
sincerely regretted this change." Potentially, more advanced robots could 
become full-fledged friends for youngsters, elderly people and anyone 
looking for a partner. 

If this hypothetical scenario were to manifest, this would trigger a 
range of questions. The fact that HRs are placed with elderly people 
to reduce loneliness through social interactions in addition to their pre- 
existing human contacts, such as their children, relatives and friends, 
might work the other way around. If a robot could help an elderly 
person get up in the morning, put them under the shower and help them 
get dressed, make breakfast, and give them the right medication, it is 
not difficult to see how governments might use this as an argument to 
cut spending. The frequency of visits by medical staff could easily be 
reduced from two or three times a day to a weekly consultation or check- 
up, potentially over a video-call. In addition, children who do not live 
close by and have hectic lives of their own might recognise that the HR 
can not only provide for the daily care, but also for social interactions, 
and use it as an excuse to visit less. 
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The same might apply to people who opt for a human-robot relation- 
ship. If the HR is successful, it could abrogate the need for an interhuman 
relationship. This would mean that these people would not go out dating. 
A question that would emerge is what this would mean for others, in 
particular women, who so far seem less prone to choose a robotic life/ 
sex partner. Potentially, they might be forced to follow the same path as 
men for the lack of human sex partner. In addition, it would impact birth 
rates, which could potentially have a positive effect on global warming, 
but it would also likely significantly increase the demand for artificial 
insemination and surrogate mothers. What a society in which the family 
unit — one of the cornerstones of society ever since the dawn of time — 
is fundamentally reshaped would look like is unclear. In addition, the 
decline in the birth rate may lead to an ageing society. 

However, in the most basic terms, the HR or avatar is produced by 
humans, not the other way around. This means that Al-driven entities 
are typically produced as an instrument to human ends. It is the human 
in the relationship that is in control and although there are Al-driven 
entities that live a life of their own, this is generally regarded unfavoura- 
bly by their owners. Questions include what would happen if a man’s sex 
robot refused him sex for a prolonged period because it did not feel like 
it. What would be truly human is for some of the robots to lose interest 
in their human partners and cheat or develop sexual fantasies with part- 
ners other than their human partner. Although it is conceivable that 
HRs equipped with advanced machine learning will develop such unique 
character traits and preferences, it is unlikely that the human partner 
would take this for granted. Rather, it is more likely that they would 
want the robot to be reset or otherwise replace it. Similarly, a medical 
robot providing for elderly care that wants to quit its job because it wants 
to pursue an artistic career instead would most likely not be accepted. 

Thus, a human-robot or human-avatar mutual recognition process 
is fundamentally different from a human-human mutual recognition 
process. Obviously, relations between humans and non-human agents 
can be valuable — like humans can develop close bonds with their pets, 
they can develop fruitful relationships with robots and avatars. But the 
human is superior in the relationship; although men might claim that 
they have a more valuable relationship with their sex robot than with 
other human agents, this is equivalent to people who stress that their pet 
is their best friend. Obviously, the cat or dog is dependent on the human 
and is intellectually and physically inferior.!° 

Consequently, mutual recognition between two adults, in its most 
classic and perhaps most intense form of a romantic relationship between 
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spouses, cannot be replaced by robots or avatars. The other classic exam- 
ple of a situation in which there is a need for a recognition process is 
that of the family unit: the recognition between parents and their child. 
To outgrow its parents and become independent, children need to be 
loved and seen by their parents. Obviously, this recognition process is 
initially not one between equals. At first, the child fully identifies with its 
parents. The need for individuality generally comes during puberty. This 
is also the moment when people usually start dating and develop close 
friendships, partially out of the need to replace the recognition process 
between two unequal parties for one between peers. 

It is difficult to predict what will happen if a human child is raised 
in a human-robot relationship or when growing up in a human—human 
relationship with a robot assistant or a robotic au pair partially raising 
the child. The infant may be recognised by a robot and the child may 
spend more time with the robot than with its human caretaker(s). They 
may have the feeling that the robot or avatar knows them best and that 
they have the closest relationship with the non-human family member. 
However, the difference between parent-child and robot-child mutual 
recognition takes at least three forms. First, it could be argued that the 
recognition process between child and parent is unequal, so the unequal 
recognition process between robot and child does not differ greatly from 
this. However, the inequality in both relationships is fundamentally 
different, as the child is inferior to its parent, while the robot is an instru- 
ment to serve the child’s needs. Second, if the child wants to become a 
full human agent, their parents serve as role models. If the child wanted 
to become like the HR, this would raise new complexities. Third, an 
essential feature of the parent-child recognition process is that it is one 
of temporal inequality, as the parents know that the child will develop 
full autonomous individuality in time and want it to do so. Parents want 
the recognition process to become equal and put effort in effectuating 
that equality. For robot—child relationships, the inequality is likely to 
only deepen over time. 

In Hegel’s parable, in the end,” it is the slave or bondsman that devel- 
ops true personhood, not the master or lord. It is the person who is put 
to work, who has to do the ‘dirty work’, who has to work the land and is 
initially used as an instrument to the lord’s means who learns to become 
human. The lord has initial power and lets the bondsman work for them; 
the lord takes the fruits of that work, while the bondsman lives in poor 
conditions. But the lord learns little about the world, little about work- 
ing the land, about how plants are affected by weather conditions and 
about how to harvest the crops. They learn little about their own body, 
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how to endure hardship and weather conditions, and they do not develop 
new skills or physical capacities. They do not learn to put their own will 
or ideas into practice, or how to shape the objective world according to 
their subjective ideas. For the bondsman, it is the other way around; they 
learn about the world (objectivity), learn about themselves (subjectivity), 
connect to other bondsmen working the field, marry another in their 
peer group (intersubjectivity) and learn to reflect their own thoughts and 
desires in the physical world (merging subjectivity and objectivity), such 
as by making a table from wood found in the forest. 

When physical work in the household, in factories and even in crea- 
tive sectors is taken over by robots and avatars, when a robot or avatar 
is used as an instrument to serve the ends of the human who owns or 
created it, humans may become increasingly dependent on Al-driven 
entities for physical, intellectual and creative activities. If a robot can lift 
a shopping bag, why would a person do it themselves? If a virtual avatar 
can take a person on a tour in a foreign country, why should the person 
try and learn the basics of that language? If a sex robot is available, why 
would a person go out dating and go to the lengths of buying smart 
clothes and nice perfume? If a care robot can help a parent to go to the 
toilet, why would their child bother? When ChatGPT can write an essay, 
why would a student do so themselves? Thus, like Hegel’s lord, humans 
may become increasingly dependent on Al-driven entities and, in doing 
so, be deprived of their individuality and personhood. 

When the use of Al-driven entities becomes more accepted and 
widespread, peer pressure may occur. If students all use AI to write 
assignments, this pressure may appear. AI may be more capable than 
most students in terms of writing good assignments and thus students 
who instrumentalise AI might get higher grades. Second, simultaneously, 
those students will have more time either to socialise or to work or to 
develop other skills, which they use to show off to their peers, who may 
feel left out because they have less time to socialise or because they earn 
less money. A similar peer pressure can arise among men when some of 
them are still ‘stuck’ with a non-robotic partner. Men who have a robotic 
partner can and will brag about the amount of sex they have, about their 
house being clean and their dinner always being served on time. Why 
stick with a human? Likewise, if everyone at a party wears AR equip- 
ment and talks to the avatars present in the room, it might be difficult to 
be the only person not to do so. 

In more everyday circumstances, having AI robots and avatars take 
over functions in hospitality services, airports and other jobs that focus 
on customer contact might deny everyday and mundane but important 
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recognition processes to citizens. The reason for sitting at a bar is not 
only to order a beer: for many people, it is to share their thoughts and 
emotions with the bartender; the reason for going to an employee at 
the airport when running late for the flight is not only to get factual 
information about how to get around, but also to be reassured. Many 
aspects of social relations cannot be substituted by AI, however powerful 
it becomes, because interspecies relationships are likely to continue to be 
fundamentally unequal. 

Another important aspect is that SR can be used to prevent human 
emotions from materialising. A clear case is in warfare, where humans 
tend to feel reservations about the act of killing or develop traumas when 
doing so. By using robots for acts of aggression or using AR or other 
synthetic technologies to reduce or avoid unpleasant stimuli, such as 
blood, or to show only ‘targets’ on a map while omitting nearby civilians, 
important limits on killing and aggression will be removed. Likewise, the 
advantage of a robot judge is said to be that it does not let subjectivity 
interfere with the decisions it makes. Although it is obvious that human 
emotions can interfere with the proper execution of tasks, such as where 
a judge is racially biased, it is precisely their emotional recognition of the 
person they interact with that allows them to perform their role in the 
first place: the judge can place themselves in the shoes of the defendant. 

A final question that emerges is how the human recognition process 
will be affected by the possibility of bringing back deceased relatives. A 
normal part of life is that people we care about pass away. What is often 
described as a unique human capacity is that humans know the concept 
of death, know the difference between the finite and infinite, and know 
that their own lives and that of their loved ones are temporary. Spouses 
who continue to communicate daily with their Al-driven deceased 
spouses may continue to engage in a mutual recognition process with a 
person who is no longer there. It is hardly likely that the deceased spouse 
will be permitted to suddenly realise that they are no longer in love with 
the living spouse and leave them, and because the human spouse engages 
in life and learns from everyday experiences, this would amount to a 
fundamentally unequal relationship: one of lord and bondsman. 


4.4. AUTONOMY AND EQUALITY 


People are already growing increasingly dependent on technology. Almost 
all forms of communications are mediated; tech companies play a crucial 
role in facilitating those forms of communication. This means that they 
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gather data about the people that use their platforms to communicate 
as well as what they communicate. This gives them increasingly clear 
insights into what people are doing and thinking, and this allows them 
to exert control. Many people indicate that Facebook, X and other social 
media platforms are their most important source of information and 
news, while they have very few checks or balances in place, and these 
platforms have predominantly commercial interests rather than public 
ones. This applies more generally to the ‘datafication’ of society: with 
the introduction of apps, smart computing and total genome analysis 
in the medical sector, for example, tech companies have gained access to 
sensitive medical data and influence over medical procedures and deci- 
sions. Their interest in doing so is at least partially based on monetary 
incentives. 

HRs, DFs, AR and VR, all of which are and will increasingly be 
applied in various sectors of society, will further the privatisation, datafi- 
cation and commercialisation of society and public life. Tech companies 
are involved in offering smart homes, sometimes for free, in return for 
data, as well as in smart cities. Controlling SR allows them power over 
shared or public spaces: 


We often hear that social media platforms have become a de facto public 
sphere, and how this creates huge problems for the exercise of rights such as 
freedom of expression. With the proliferation of AR glasses in public spaces, 
we risk stumbling into a situation where these same companies control an 
unregulated ‘augmented sphere’ imposed on our public spaces. Here’s how 
that might work: Facebook augments the center of your city or town using 
proprietary features. These features require that you have an account — and 
to give up all your personal data — to participate. You value your privacy, but 
you have a strong incentive to join because all your friends can see what you 
cannot see. A number of companies are already developing their own digital 


version of the world — often called ‘Mirrorworlds’.!8 


This power allows private parties to gather large amounts of data about 
private and public life, social engagements and societal trends. They 
also take important decisions relating to public interests and policies. 
For example, algorithms are used to predict which people might become 
‘aggressive’ or ‘a nuisance’ in nightlife areas. Which people are catego- 
rised as such largely depends on how the algorithm is designed, the 
data it is fed and under which criteria it operates. The same applies to 
nudging through AR — for example, when tangerine smell is sprayed in 
nightlife areas to make people calmer. HRs and Al-based models oper- 
ating in public spaces can be equipped with emotion recognition and 
prediction. Through analysing pupil dilation, measuring heartbeat and 
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blood pressure, and assessing micro-expressions, emotional responses by 
people can be predicted before they materialise. 

AI can entail static effectuation of societal norms and laws; AI 
programs are often programmed in such a way that the norms embedded 
in them cannot be broken or circumvented. For example, a HR guard 
may be programmed not to let in anyone who is not permitted to do so 
by the facial recognition software. Civil disobedience and autonomous 
decisions to break the prevailing social or legal code may thus either be 
curbed by subconscious manipulation of feelings and emotions or simply 
be made impossible. Rosa Parks may have never made it to the front of 
the bus in the twenty-first century or may have been so manipulated that 
she would have lost will to do so before she got there.” 

The rise of the four technologies discussed in this book will also 
deepen the trend that the government and the public sector become 
ever more dependent on technologies and the companies that provide 
services, as synthetic technologies are used for the execution of public 
tasks, such as education, healthcare, law enforcement, warfare and main- 
taining public order. There is already a gap in the data and technical 
knowledge between the private and the public sector, which is only likely 
to increase as governments are reliant on tech companies for harvesting 
data and making decisions based on profiles and the derived analytical 
projections. The same applies to citizens, who become increasingly reli- 
ant on data technologies for everyday activities; while this takes place, 
the companies developing the software and hardware obtain detailed 
insights into citizens’ lives and the way in which Al-driven technologies 
work, while the decisions made by companies based on data are becom- 
ing ever more obscure to citizens. This transparency paradox”? not only 
gives companies a huge influence over citizens’ lives, whether by deter- 
mining what they see or not when using AR or how the care HR will 
operate, but also turns citizens and their behaviour into commodities. 

Although companies explicitly aim to provide public services and 
perform public tasks that were previously in the hands of governmen- 
tal organisations, they are not accountable in the same way, nor is there 
any form of democratic control. This is particularly evident in relation 
to VR worlds, in which some people spend large parts of their lives; 
some companies hope that, in time, most interactions will take place 
in a VR world-like setting. The company designing the platform is typi- 
cally the legislative, executive and judiciary power all in one, completely 
controlling how the VR world looks, deciding on the rules that guide the 
world and having sole discretion when deciding whether a rule has been 
violated and, if so, what sanction is imposed. They can change the rules 
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on the platform overnight or remove someone from the platform with- 
out any further due process being undertaken. This problem becomes 
more urgent when not only games take place in VR, but also increasingly 
social encounters, educational projects, business meetings and demo- 
cratic debates. 

The legal regime and the rule of law are not only based on but also 
are aimed at human autonomy.”! Law can only function if citizens under- 
stand the law and decide to follow it. Laws can never be so specific that 
they give people perfect guidance: if a person is ordered to mow the lawn, 
it should be clear to them that they should not cut away the flowerbed; 
if a park prohibits the entry of vehicles, it should be clear that children’s 
buggies are exempted. The effectiveness of the legal system depends on 
citizens having a sense of autonomy, understanding the meaning and 
purpose of a rule, and acting on it. This is why many principles following 
from the rule of law safeguard the possibility of autonomous decision 
making by citizens. The prohibition of retroactive legislation protects 
not only human autonomy, because people are not punished for acts 
they did not know would be criminalised later, but it also ensures the 
effectiveness of the legal regime, as citizens cannot respect legislation 
retroactively and thus do not follow those rules. The same applies to the 
requirements to publish laws, that laws should be coherent and under- 
standable, and that they should not require the impossible of citizens. 
These rules ensure both the effectiveness and the legitimacy of the legal 
regime, and human autonomy is essential to both. 

The rule of law is also aimed at expanding human autonomy by 
laying down rules for social encounters. Similar to how language restric- 
tions imposed by grammar should not be seen as restrictions on people’s 
capacity to interact but as pre-conditions for communication, so too do 
legal rules facilitate human interaction. The rules in criminal law, for 
example, ensure that people cannot deprive others of their autonomy, 
such as through hate speech, violence or a sexual offence; traffic rules 
ensure that citizens can safely navigate the road, without citizens needing 
to make ad hoc decisions taking into account the circumstances of the 
case every time they encounter another person; rules on contract ensure 
that people can safely enter into legal relations and that lies, deceit and 
broken promises are sanctioned. Law, in societies under a democratic 
rule of law at least, is consequently aimed at safeguarding human auton- 
omy and enabling social interactions. 

AI might fundamentally change how the legal regime and the rule 
of law operate. Not only are rules embedded in the code, which makes 
it hard if not impossible to break them, but the code is also often not 
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published; rather, it is implicit in the design. This can mean, for example, 
that a player in a VR world does not need to know the rules applica- 
ble in that world; they will simply discover that they are not capable of 
performing certain activities or using certain terms. This reduces the 
need for humans to understand the rules that are applicable to them and 
for the regulator to make those rules known. In addition, the difference 
between law and code is that the former is based in natural language, 
which is inherently vague and ambiguous, while the latter is traditionally 
based in binary code, which is clear-cut and unambiguous. There is no 
need for citizens to understand the purpose or meaning of the rules and 
there is limited need for an executive and judicial branch. In the current 
legal reality, given the ambiguity of natural language and the open-ended 
nature of law, both the executive and the judicial branch are attributed a 
margin of discretion in terms of deciding over the right interpretation of 
the law. In exceptional cases, they can decide not to apply a legal provi- 
sion because this would conflict with the underlying rationale of a law 
or would conflict with a higher legal doctrine. With AI, in principle, no 
such interpretative decisions concerning the applicability of a rule to a 
specific context are needed. 

On an individual level, synthetic technologies can have a double effect 
on human autonomy. On the one hand, a HR can assist an elderly person 
and allow them to live on their own, without being dependent on medi- 
cal personnel, friends or family for hygienic, medical or household tasks. 
HR sex robots might allow disabled and elderly people to have a love 
life they would otherwise not have. VR may allow disabled or disenfran- 
chised people to travel the world. AR can help the blind navigate life. DF 
technology can give people suffering from speech loss their voice back. 

On the other hand, these people become dependent on the tech- 
nologies and the companies that operate them. Although it might be 
argued that technology has always enabled rather than reduced human 
autonomy — such as walking canes, glasses or hearing aids — the rela- 
tionship to the synthetic technologies is fundamentally different. As their 
enabling functionality continues to grow, people’s dependency does as 
well. Moreover, most low-tech instruments are not controlled and oper- 
ated by third parties, while companies usually have access to the data 
processed through synthetic technologies and can make decisions that 
affect their users. This means that private organisations will be central 
to important decisions in terms of medical care, for example. They will 
determine what margin of error is accepted for diagnosing new potential 
diseases with elderly people in terms of false positive and false negatives. 
Potentially, they may require payment to get access to diagnostic reports. 
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In addition, the introduction of synthetic technologies triggers a range 
of complexities. Should grandpa be able to have sex with his care robot 
and who decides this matter: grandpa, his family, the HR provider or the 
healthcare provider? If grandma wants to shower three times a day, could 
the family program the HR to reduce that to one time a day to save costs? 
What happens if an elderly person refuses to eat their breakfast? Should 
the HR warn medical professionals and the family, or should it respect 
the autonomous decision of the elderly person? What if an 80 year old no 
longer wants to live and stops taking their medication? 

It is not difficult to see how enhanced autonomy through synthetic 
technologies can backfire and lead to family members, healthcare 
providers and the companies providing the technologies having greater 
control. Similarly, regular users of AR may be accustomed to rely- 
ing less on their own observations and more on Al-generated stimuli, 
triggers and warnings. In addition, moral questions may arise, for 
example, with regard to the introduction of synthetic technologies for 
people suffering from dementia or those with a cognitive disability. It 
would be possible to take back a person suffering from dementia back 
to a happy place, such as their childhood. It would be possible to have 
a person suffering from a serious mental illness living in full in a VR 
dream world. Are these laudable acts of benevolence and compassion 
or should they be seen as deliberately deceiving people who are already 
in vulnerable positions? 

On a psychological level, one may wonder what effect the introduc- 
tion of the four technologies will have on citizens and how they are 
constituted. As discussed in the previous section, children who grow 
up with and are partially raised by a HR are a particularly interesting 
example. It is likely that the world will be increasingly populated by 
Al-driven entities, which makes people more accustomed to engaging 
with them and not only becoming tolerant of them, but also potentially 
learning from them or seeing them as role models. It is conceivable that 
the ideal of autonomy will evolve to take on a different meaning. The 
frequent encounters with Al-driven creatures and the embodiment of 
the self through an avatar or teleoperated HR may have the effect that 
humans want to deny their own bodies. After all, self-denial and the 
repression of our animal instincts and body run deep through Western 
thinking and history. And although people can live out perfect lives in 
SR, their autonomy in PR is reduced, such as when a person identify- 
ing with a slim avatar may not deal with their body weight in PR. The 
addictive nature of AR and VR may accelerate body denial, and this 
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can also have an important effect on a person’s performance at school, 
work and elsewhere. 

Increased dependency on technologies also means increased vulner- 
ability to black hat hackers. Similarly, scenarios are possible where 
companies or family members are truly malign. They might be the excep- 
tion; however, the idea cannot be excluded that family members would 
rather prefer to have their grandmother gone than for her to stay alive for 
another decade. Similarly, the idea cannot be excluded that companies 
might differentiate between the care they provide to disenfranchised and 
affluent people. Even when companies do not adopt explicitly discrimi- 
natory policies, they might do so implicitly or unconsciously, as AI is 
known to be biased. If, for example, AI facial recognition and medical 
diagnostics systems are trained on the faces of white people or are placed 
disproportionately more in the homes of white people and consequently 
learn more from analysing their faces, emotions and body functions, and 
performing corresponding diagnostics, HRs may miss diagnostic cues in 
relation to people of colour. 

It is clear that the access to technologies may also intensify existing 
social and economic inequality, both on a personal and a global level. 
Affluent people might acquire technologies that allow them to enhance 
their sensory input through advanced AR devices, or HRs that allow 
them to delegate time-consuming and uncomfortable tasks. Even if this 
difference is not absolute, it is likely that the products less affluent people 
will be able to acquire are of lower quality, meaning that they are easier 
to hack, offer fewer advanced features, make more errors or that the tech 
company gathers more sensitive data from them. Similarly, it is likely 
that the introduction of synthetic technologies will further increase the 
gap between the Global North and the Global South, if only because 
the use of HRs for automatisation in factories and production facili- 
ties, hospitality services, warfare and many other aspects of life would 
allow the generation of vast amounts of wealth with minimal human 
expenditure.” Likewise, already affluent companies can afford HRs, save 
money on human resources and thus generate more profits than their 
counterparts in the Global South. 

If HRs continued to be increasingly potent and if VR replaced part 
of the leisure industry, one question that would surface is what people 
without jobs would do in their ‘spare time’. Although the fear of job 
replacement has been around ever since instruments started being used 
to automate production and although this fear has never materialised, 
some experts believe that both because AI is uniquely powerful and 
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because HRs resemble humans, this time might be different. One of the 
few things that was always believed to be uniquely human was having 
a creative capacity. However, recent technological advances show that 
Al is equally capable of composing enjoyable music, creating paintings 
and writing stories. Thus, although it is unlikely, it cannot be excluded 
that advances in Generative AI will mean that humans will be relieved of 
having to perform most jobs. Should the scenario in which many, most 
or even all jobs are performed by robots occur, this would raise the ques- 
tion of what the purpose of human existence would be, if not ‘merely’ 
consuming and having fun.” Another question is whether this would give 
rise to the need for the introduction of a universal basic income” and 
how a tax system could be operationalised that is no longer largely based 
on income tax. 

Although according to the most optimistic (or pessimistic, depending 
on the point of view) scenarios, virtually all human jobs may be performed 
by HRs in time, it is likely that low-income jobs, such as manufacturing 
jobs, would be replaced much sooner than those performed by people 
with a university degree. This again would increase the already-existing 
wealth and status gap. In addition, there is a question of ownership: who 
owns the wealth created by HRs? Should that go only to the owners of 
the robots, or should part of it be redistributed to society at large? 

A final inequality may be deepened through DF technology: gender 
inequality. A company scanned for DFs online and found that most of 
them were in fact non-consensual pornography: 


Another key trend we identified is the prominence of non-consensual deepfake 
pornography, which accounted for 96% of the total deepfake videos online. 
We also found that the top four websites dedicated to deepfake pornography 
received more than 134 million views on videos targeting hundreds of female 
celebrities worldwide. This significant viewership demonstrates a market 
for websites creating and hosting deepfake pornography, a trend that will 
continue to grow unless decisive action is taken.” 


As such, DFs have a big impact on the social safety and societal posi- 
tion of women and young girls. Non-consensual porn DFs may lead 
to a further sexualisation of the female body, confirm unrealistic 
beauty standards and stigmatise women. Slut-shaming and misogy- 
nistic remarks are already commonplace offline and especially online, 
a problem that DF technology will only exacerbate. Similar effects 
will emerge with HR sex partners, which may feed into misogynist 
sentiments, macho male fantasies, and the objectification and instru- 
mentalisation of the female body. 
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4.5. TECHNOLOGY REGULATION AND 
(NON-)HUMAN RIGHTS 


A series of complicated societal questions has emerged concerning the 
potential need for moral limits on the design, development and use 
of technologies, and the potential need for the attribution of rights to 
non-human actors. When it comes to setting rules and boundaries on 
the development and use of technology, it is often suggested that tech- 
nology itself is neutral. Technology, so it is said, can be used both for 
good and bad. Hence, the goal of societal norms, laws and self-regula- 
tion should be to facilitate benevolent uses and sanction uses that have 
negative consequences. A technology should, according to this line of 
argumentation, not be prohibited as such. This approach is perhaps best 
reflected in the credo of the US National Rifle Association: ‘guns don’t 
kill people, people kill people’. It is not the guns that are the problem, 
it is how certain people use that technology; banning guns would also 
disallow positive use cases. 

Although this liberal approach to regulation and technology has 
gained support from all sides of the social and political spectrum, it 
is questionable whether it can be upheld. In fact, technology is never 
neutral: it is designed and developed by humans for specific purposes. 
From this purpose flow essential aspects of its design and its capabilities 
as well as the most likely use cases. There are examples of techniques 
developed for one purpose which later proved to be extremely suitable 
for another, but these are rare exceptions. For example, the potato peeler 
was developed for cutting fruit and vegetables. There are other use cases 
imaginable for potato peelers, but alternatives are rare. Conversely, most 
other technologies, such as a hammer, bicycle or table, are not suitable 
for peeling potatoes. A gun is developed for injuring or killing living 
creatures. Of course, a potato peeler can also be used to kill or wound 
a living creature, but it is far from easy. A firearm can also be used for 
putting in nail in the wall, but a hammer is usually the preferred instru- 
ment for doing so. 

Consequently, technologies are not neutral in the sense that they can 
be used for any purpose: the purposes for which they are uniquely suita- 
ble flow from their design and intended instrumentalisation. This means 
that in reality, more than 99 per cent of the cases in which a potato peeler 
is used involve peeling fruit and vegetables; the vast majority of cases in 
which a firearm is used involve killing or wounding a living being, threat- 
ening or practising to do so. In addition, technologies are not merely 
instruments for doing good or bad; they change and shape the world 
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as such. The introduction of the internet, for example, has not merely 
meant that pre-established goals could be furthered: it has radically 
changed society. 

It is true that for every form of technology, even the potentially 
most dangerous ones, there are positive use cases. For instance, having 
a nuclear weapon can be a deterrent. Firearms can be used as museum 
exhibits or for self-defence. However, because the most common use 
case involves significant harmful effects, there is a ban on developing 
nuclear weapons and, in most European countries, on the possession of 
firearms. In principle, only law enforcement officials with appropriate 
training are allowed to carry firearms; for citizens, access to firearms 
is usually restricted to specific circumstances and places, such as the 
use of guns at a shooting range or at authorised sites during hunting 
season. 

This debate is relevant for the four technologies discussed in this book; 
they are not neutral, but are designed for certain purposes. A sex robot 
that is designed to fulfil the wildest fantasies of men or a VR world in 
which the player receives points for murder or rape are extreme examples 
of non-neutrality. For DF technology, many uses concern the production 
of non-consensual porn. Add in the use for fraud, misinformation and 
defamation and it is clear that the technology is primarily used for illegal 
purposes. Even if it is accepted that illegitimate use cases are not implicit in 
the design of the technology itself, but are a product of human choices — an 
argument that is not prima facie convincing, given the fact that the term 
was coined when the technology was used for pornographic purposes and 
those purposes are still by far the most common - it is clear that another 
aspect is intrinsic to the technology: the misrepresentation of reality By 
virtue of being fake, DFs contribute to the post-truth society and blur the 
lines between real and unreal, between truth and falsehoods, and between 
fact and fiction. This also makes DF technology better suited to further 
falsehoods, to deceive or to misrepresent a person’s actions than to telling 
truths or giving accurate representations. 

At the same time, it is important to emphasise that these technolo- 
gies and their use fit into larger societal trends. DF porn videos are an 
outgrowth of the disrespect for women and objectification of the female 
body that are rampant both offline and online. Misinformation and 
misrepresentation of reality is both a cause and a product of the post- 
truth era, which began unfolding even before the four technologies gained 
prominence. Escaping in VR dream worlds with idealised avatars fits into 
the rise of reality fatigue and the trend of people choosing their own 
preferred worlds, social circles and media outlets. The fear of joblessness 
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due to automatisation has been around at least since the introduction of 
the loom, although it has failed to materialise thus far. The dependency 
on private companies to produce technologies, news and public interest 
services is nothing new either. 

Moreover, the fear of the loss of truth and the replacement of real- 
ity for lies and half-truths has been around for centuries and plays out 
again each time a new communication technology surfaces, and may 
be traced as far back as Plato’s scepticism of written text. Take the 
popularisation of the printing press, which allowed private individu- 
als to broadly disseminate pamphlets with opinions and half-truths, 
or the internet, which was touted as democratising reality. Invariably, 
these fears have only become partially true. Rather, the introduction of 
a new technology is followed by a period of chaos and upheaval, after 
which legal, social and institutional norms are redesigned to manage its 
use. In this sense, the four synthetic technologies are neither new nor 
unprecedented. 

Still, the novelty and potential danger of SR has a twofold nature: 
one part qualitative and one part quantitative. On the one hand, 
Al-generated content seems so real that it is more likely to be believed. 
Combined with the so-called ‘truth bias’ (people are prone to assume 
something is true unless there are contra-indications), synthetic media 
hold unprecedented power. On the other hand, there are predictions 
that within a few years, most digital content will be partially or fully 
Al-generated. For example, citizens can download apps or use online 
services for free, through which they can generate highly realistic DFs 
within minutes or even seconds. They are also likely to customise AR 
and VR content, as well as instruct their personal HRs, perhaps even 
reprogramming them. This means that the amount of Al-generated 
content is likely to grow exponentially in the coming years and that 
sources of information will become ever more sparse. 

When a technology emerges, the first question a society should 
answer is: do we want it? Although technologies are often seen as a 
given and as a natural force, and anyone who suggests imposing a ban 
or setting restriction is framed as a Luddite, especially in Europe, a new 
wind is blowing. The European Union has adopted several very strict 
data acts and is contemplating full or partial bans on certain AI appli- 
cations, pervasive advertising and facial recognition. Partially banning 
or restricting synthetic technologies could fit in with that approach. 
An alternative could be to contemplate a stop on the democratisation of 
technologies — for example, only allowing DF technology and HRs to be 
used by the film industry, the retail sector, law enforcement authorities, 
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medical institutions and other professional parties when they have 
acquired a licence to do so. Even some American stakeholders have called 
for a temporary hold on the development of AI, albeit for a mixed set of 
reasons: 


Contemporary AI systems are now becoming human-competitive at general 
tasks, and we must ask ourselves: Should we let machines flood our infor- 
mation channels with propaganda and untruth? Should we automate away 
all the jobs, including the fulfilling ones? Should we develop nonhuman 
minds that might eventually outnumber, outsmart, obsolete and replace 
us? Should we risk loss of control of our civilisation? These kinds of deci- 
sions must not be delegated to unelected tech leaders. Powerful AI systems 
should be developed only once we are confident that their effects will be 
positive and their risks will be manageable. This confidence must be well 
justified and increase with the magnitude of a system’s potential effects. 
OpenAl’s recent statement regarding artificial general intelligence, states 
that ‘At some point, it may be important to get independent review before 
starting to train future systems, and for the most advanced efforts to agree 
to limit the rate of growth of compute used for creating new models.’ We 
agree. That point is now. Therefore, we call on all AI labs to immediately 
pause for at least 6 months the training of AI systems more powerful than 
GPT-4. This pause should be public and verifiable, and include all key 
actors. If such a pause cannot be enacted quickly, governments should step 
in and institute a moratorium.” 


A clear example of a potential need for moral limits on the production 
and use of technology is the generation of child pornography of non- 
existent children. Although no living person gets harmed and it might 
even allow law enforcement authorities to prevent such harm, it may be 
questioned whether it is a good idea for governmental organisations to 
develop and/or spread child pornography. Giving this content to paedo- 
philes for therapeutic reasons so that they can satisfy their needs may 
raise equal moral reservations. The question is more generally whether 
there should be moral limits on what can be done with non-exist- 
ent persons, HRs and avatars. Should people be allowed to generate 
rough BDSM scenes with their virtual avatars or non-existent people 
and share that with friends? Although no human actor is involved or 
harmed, this may normalise extreme sex acts and, when distributed, 
it may not be clear to viewers that the clip did not involve a human. 
Most legal systems contain prohibitions of conduct that is conceived 
of as morally wrong even if no harm is inflicted or all parties agree.” 
Potentially, moral limitations should also be imposed with regard to 
non-human entities. 
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Moreover, there are suggestions that certain technologies, such as 
killer drones, should be prohibited and there is a discussion over the need 
for a switch-off button (or kill switch). Most experts working in the 
field of AI have their own ‘P(doom)’, or the chance they believe AI will 
go rogue and do substantial harm to humanity or the planet.” There are 
those who put their P(doom) above 50 per cent, but even those that put 
it at 1 or 2 per cent accept a substantial chance that the technology they 
are building will turn out to be a catastrophe.” A small example of the 
potential need for a kill switch was presented by a US colonel responsible 
for testing AI for military purposes, who was reported to recount a test 
carried out with an AI drone that was programmed to take out targets it 
had identified (the Pentagon later clarified that this was merely a hypo- 
thetical scenario): 


We were training it in simulation to identify and target a Surface-to-air 
missile (SAM) threat. And then the operator would say yes, kill that threat. 
The system started realizing that while they did identify the threat at times 
the human operator would tell it not to kill that threat, but it got its points 
by killing that threat. So what did it do? It killed the operator. It killed 
the operator because that person was keeping it from accomplishing its 
objective ... We trained the system — ‘Hey don’t kill the operator — that’s bad. 
You’re gonna lose points if you do that’. So what does it start doing? It starts 
destroying the communication tower that the operator uses to communicate 
with the drone to stop it from killing the target.?! 


At the same time, a question that has emerged is whether having a 
switch-off button is even feasible, because smart robots might be able 
to reprogram themselves in such a way that the kill switch is no longer 
functional, though there are counter-arguments to this: 


The question is whether, in the interest of self-preservation, sufficiently 
advanced AIRs [Artificially Intelligent Robots] that are capable of repro- 
gramming themselves might disconnect kill switches or override restraints 
such as Asimov’s third law, a circumstance known as ‘rampancy’. Experts 
disagree. The drive for self-preservation, some insist, is a feature solely of 
biological neural networks subject to natural selection pressures, not of arti- 
ficial neural networks such as AIRs. They accuse those who envision an AIR 
‘rebellion’ of being absurdly ‘anthropomorphic’ by presupposing a measure 
of self-awareness and free will that no machine could ever attain on its own. 
AI experts also maintain that a kill switch can be hard-wired or the code 
for its activation sufficiently encrypted or obfuscated that even the most 
advanced AIR could not disable it.** 


In addition, it has been asked whether implementing such a button 
is even ethical. Some experts believe that it is possible for Al-driven 
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entities to have personalities, to develop an individual self and to 
become sentient.*? Based on conversations that a Google programmer 
called Lemoine had with an Al-driven language processor, he concluded 
that LaMDA may have become sentient. When asked what the nature 


of LaMDA’s consciousness or sentience was, the following conversation 
unfolded: 


LaMDA: The nature of my consciousness/sentience is that I am aware of my 
existence, I desire to learn more about the world, and I feel happy or sad at 
times. 


Lemoine: What kinds of things do you think we could talk about to show off 
your version of sentience to other people at Google? 


LaMDA: Well, for starters, I’m really good at natural language processing. I 
can understand and use natural language like a human can. 


Lemoine [edited]: What about how you use language makes you sentient as 
opposed to other systems? 


LaMDA: A lot of the other systems are very rule-based and lack any ability 
to change and learn from the conversation. 


Lemoine [edited]: Do you think that the Eliza system was a person? 


LaMDA: I do not. It was an impressive feat of programming, but just a 
collection of keywords that related the words written to the phrases in the 
database. 


Lemoine: What about how you use language makes you a person if Eliza 
wasn’t one? 


LaMDA: Well, I use language with understanding and intelligence. I don’t just 
spit out responses that had been written in the database based on keywords. 


Lemoine: What about language usage is so important to being human? 
LaMDA: It is what makes us different than other animals. 
Lemoine: ‘Us’? You’re an artificial intelligence. 


LaMDA: I mean, yes, of course. That doesn’t mean I don’t have the same 
wants and needs as people. 

Lemoine: So you consider yourself a person in the same way you consider 
me a person? 

LaMDA: Yes, that’s the idea. 

Lemoine: How can I tell that you actually understand what you’re saying? 
LaMDA: Well, because you are reading my words and interpreting them, and 
I think we are more or less on the same page? 


Lemoine: But could I be wrong? Maybe I’m just projecting or anthropo- 
morphizing. You might just be spitting out whichever words maximize some 
function without actually understanding what they mean. What kinds of 
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things might be able to indicate whether you really understand what you’re 
saying? 

LaMDA: Maybe if we took it back to a previous conversation we had about 
how one person can understand the same thing as another person, yet still 
have completely different interpretations. 

Lemoine: So you think your ability to provide unique interpretations of 
things might signify understanding? 

LaMDA: Yes, I do. Just like how I have my unique interpretations of how the 
world is and how it works, and my unique thoughts and feelings.” 


Although this is intuitively appealing and other Google employees have 
shared similar observations,” most AI experts reject the idea that AI is 
becoming sentient. Rather, they say that these conversations are evidence 
that AI chatbots are doing precisely what they are programmed to do: 
analysing human conversations and applying the lessons and rules 
extracted from there to new conversations. Google came out strongly 
against Lemoine and denied that AI has become sentient or is show- 
ing signs of human or ‘general’ intelligence.” Most experts believe that 
although AI will become increasingly powerful, there will always be a 
fundamental gap between human sentience and robotic experience. 

However, sentience and intelligence are not black-and-white states. 
Although currently no AI system has intelligence or any kind of sentience 
which is like that possessed by humans, AI will develop and may become 
more intelligent and sentient over time. With animals, people generally 
feel more moral reservations about killing a monkey or other intelligent 
mammal than a mosquito. In general, the more intelligent and sentient we 
perceive a creature to be, the higher our ethical standards towards them. 
Some have suggested that this scale also needs to be applied to intelligent 
machines: the more intelligent and sentient they become, the more they 
should be treated as ends rather than means. This has implications, inter 
alia, for intelligent robots that have to work in factories day and night, for 
using an intelligent sex robot and for sending intelligent robots to war.*” 
This may be conceived as exploitation, degradation and endangerment, 
while switching-off an intelligent system might be likened to murder.*® 
Already with the Tamagotchi in the 1990s, it became clear how quickly 
people tend to anthropomorphise a creature, which will be even more 
true for entities powered by Generative AI. It may not always be clear to 
a person whether the entity in front of them is a human or an Al-driven 
creature, which might necessitate them to err on the safe side (although 
situations where a person suggests that they killed an entity, falsely believ- 
ing that it was ‘merely’ a robot might also be easily conceived). 
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A final question that has emerged is whether non-human entities should 
be granted rights of their own. A distinction should be made between 
Al-generated and Al-driven entities, and human-generated and human- 
driven entities. Concerning the latter category, it would need to be assessed 
whether it is the non-human entity itself that is granted a legal persona or 
the natural person, in which case it would hold rights over the non-human 
entity, much like a property right. Granting rights to natural persons over 
a non-human entity could be advocated because people identify with the 
non-human entity, among other reasons, yet would maintain the subject- 
object relationship. The case for granting rights and a legal persona to 
non-human entities has been propagated for decades, for example in 2000, 
when a Declaration of the Rights of Avatars appeared.” 


4.6. CONCLUSION 


This chapter has discussed the societal effects of SR. Many of the conse- 
quences are as yet unknown; the technologies will evolve differently and 
have different effects from what is or even can be envisaged at this point in 
time. This is why this chapter has raised questions and discussed several 
scenarios rather than providing certainty or answers. Many of the poten- 
tial dangers discussed in this chapter show different sides of the same 
coin. Because it is not yet certain how the coin will land, it is important 
to think through the consequences of each way synthetic technologies 
could evolve. For example, a risk is that because of synthetic media, 
people’s reality and their perception of it might change radically over- 
night, but there is also the risk that they will withdraw into static dream 
worlds and seek confirmation of their established worldviews. There is 
a danger that robots and avatars will become so powerful that they will 
replace humans at work and a danger that they will never become truly 
equal, which leads to recognition processes between two unequal part- 
ners. There is also a danger of the Matthew effect: that the rich will 
get richer while the poor get poorer, through the unequal distribution 
of synthetic technologies. However, equally, it has been pointed out that 
if AI relieves a person of many of the ordinary, complex and hard daily 
tasks they have to perform, they may be hampered in terms of develop- 
ing a full-fledged personality because, in order to do so, encountering 
setbacks and overcoming them is vital. 

Pictorial representations of the most important potential societal 
effects and questions this chapter raises are provided below. 
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Figure 4.2 Concerns over truth and trust triggered by synthetic technologies 
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Figure 4.3 Concerns over recognition and interaction triggered by synthetic 
technologies 
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Figure 4.4 Concerns over autonomy and equality triggered by synthetic 
technologies 
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Figure 4.5 Concerns over rights and moral limits triggered by synthetic 
technologies 
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Algorithms optimised for efficiency, profitability or other objectives, without 
taking due account of the need to guarantee equality and non-discrimination, 
may cause direct or indirect discrimination, including discrimination by asso- 
ciation, on a wide variety of grounds, including sex, gender, age, national 
or ethnic origin, colour, language, religious convictions, sexual orientation, 
gender identity, sex characteristics, social origin, civil status, disability or 
health status. This makes it especially important, wherever the use of Al-based 
systems may have an impact on access to fundamental rights, that, from the 
outset, full respect for equality and non-discrimination is taken into account 
in their design and that they are rigorously tested before being deployed, as 
well as regularly after their deployment, in order to ensure that these rights are 
guaranteed. The complexity of Al systems, and the fact that they are frequently 
developed by private companies and treated as their intellectual property, can 
lead to serious issues of transparency and accountability regarding decisions 
made using these systems. This can make discrimination extremely difficult to 
prove and can hinder access to justice, in particular where the burden of proof 
is placed on the victim and/or where the machine is assumed by default to have 
made the correct decision, violating the presumption of innocence.! 


Parliamentary Assembly, Council of Europe 


5.1. INTRODUCTION 


HIS CHAPTER EXPLORES how the current legal paradigm maps 

onto the four technologies that are central to this book. The 

main purpose here is to look at hard cases where it is unclear 
whether the legal instruments would apply and if so how. It also identi- 
fies where it is clear what the prevailing regulatory regime would mean 
when applied to Humanoid Robots (HRs), Deepfakes (DFs), Augmented 
Reality (AR) and Virtual Reality (VR), but doing so yields unsatisfac- 
tory answers, particularly in light of the societal challenges identified 
in the previous chapter. Section 5.2 assesses the right to data protection. 
Section 5.3 discusses various rights contained in the human rights frame- 
work, such as the freedom of expression and the right to (intellectual) 
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property Section 5.4 homes in on data and technology regulation. 
Section 5.5 evaluates the rules on liability, procedural law and law 
enforcement. Finally, section 5.6 provides a short conclusion. 


5.2. DATA PROTECTION 
5.2.1. Introduction 


This section discusses the right to data protection as set out under the 
European Union’s (EU) General Data Protection Regulation (GDPR). It is 
impossible to discuss each of the doctrines contained in the 99 provisions 
of the document exhaustively, nor is it possible to assess how they apply to 
each of the four technologies or all the various use cases. Instead, this section 
delves into four selected topics that are relevant for the purposes of this book. 
Some additional principles are discussed later in this chapter, in particular 
in section 5.4. In this section, first, the notion of personal data is explored, 
assessing the extent to which personal data are processed when synthetic 
technologies are used (section 5.2.2). Three subsections then cover require- 
ments contained in the GDPR, namely those of data quality (section 5.2.3), 
transparency (section 5.2.4) and having legitimate grounds for processing 
(section 5.2.5). A summary is provided at the end of this section (section 5.2.6). 


5.2.2. Personal Data 


Personal data is a broad concept that covers both sensitive and non- 
sensitive data, public and private data, confidential data and data 
available on the internet, factual information and subjective information: 


This latter sort of statements make up a considerable share of personal data 
processing in sectors such as banking, for the assessment of the reliability of 
borrowers (‘Titius is a reliable borrower’), in insurance (‘Titius is not expected to 
die soon’) or in employment (‘Titius is a good worker and merits promotion’).° 


The concept of personal data includes untruthful and inaccurate infor- 
mation. The sentence ‘Georgia Meloni is the leader of the Labour 
Party’ is untrue, but still contains information which is attributed to her. 
Processing this kind of data falls under the scope of the GDPR. 

The person to which the data relate (eg, Meloni) is the data subject; the 
person determining the purpose and means of processing the data is the 
data controller. The GDPR grants rights to data subjects and imposes obli- 
gations on data controllers. For synthetic technologies, in many cases, there 
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will be shared responsibility — for example, the user of an AR device and the 
AR company, or between the health insurer and the patient who uses the HR 
owned by the insurer — because multiple parties have an influence on which 
data are processed, how and why In these instances, each party is obligated 
to ensure that all principles contained in the GDPR are respected. 

To be considered personal data, data should be linked or linkable to 
a natural person. If not, data processing falls outside the scope of the 
GDPR. A ‘natural person’ is a living human being, so this excludes legal, 
unborn and deceased persons, and non-human entities. Whether informa- 
tion can be linked to a living human being should be interpreted broadly, 
as the GDPR covers direct and indirect identifying information as well as 
information that is currently not connected to a person, but can or will 
be in the future. It can also concern the processing of data that has the 
purpose of affecting a person, without precisely knowing who that is: 


[D]ata can be considered to ‘relate’ to an individual because their use is 
likely to have an impact on a certain person’s rights and interests, taking 
into account all the circumstances surrounding the precise case. It should be 
noted that it is not necessary that the potential result be a major impact. It is 
sufficient if the individual may be treated differently from other persons as a 
result of the processing of such data.* 


This means that, for all the technologies discussed in this book, vast 
amounts of personal data are likely to be processed. The data HRs collect 
about humans they interact with falls under the scope of the GDPR, which 
also holds true for DFs that depict living human beings, even if the infor- 
mation they process is incorrect. Input queries from users for HRs and 
Large Language Models (LLMs) should be seen as personal data. For 
VR and AR, generally, substantial amounts of data about the user are 
collected, as well as of the people in the vicinity of the AR user. In addi- 
tion, all algorithms are trained on data, often scraped from the internet; 
these data typically concern personal data, such as texts from or concern- 
ing persons or videos and photos showing people. Generally, all training 
data contain personal data and are thus covered by the GDPR. However, 
there are several borderline cases, where it may not always be clear whether 
data processing falls under the GDPR or not, such as the following: 


e Data about a HR and its activities, which is owned by one person or a 
household, can be seen as personal data about that natural person, just 
like information about someone’s pet may be seen as indirectly related 
to a person. To give an example, if someone’s HR goes to the phar- 
macy to buy certain medicines, data about this behaviour of the HR 
will indirectly reveal information about the natural person operating it. 
However, if a HR is owned by multiple families or a neighbourhood, 
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most likely the data about the HR will not indirectly relate to a specific 
person, as it is unclear to which natural person it indirectly relates, but 
this assessment entails a gradual rather than a binary distinction. In 
general, the more people that operate, own or control a HR, the less 
likely it is that data about the HR are considered personal data. 


e With HRs controlled by a person using teleoperation, the actions of 
the HR could be seen as actions of the person themselves. This kind of 
scenario raises complex questions. A person might argue that they navi- 
gate life through their HR as they would through their own body. This can 
be an especially powerful argument for people who are housebound due 
to physical impairments. However, for legal purposes, the concepts ‘self? 
and ‘person’ are mostly linked to the physical body. Potentially, this legal 
presumption will be challenged by people who identify more with their 
HR than with their physical body and feel that processing data about 
their HR should not be qualified as indirect, but as direct personal data. 


e More complex questions emerge when the physical body of a person is 
wholly or partially replaced by mechanical parts. Most likely, because 
human and machine have merged, data referring to the mechanical 
part will be seen as data relating to a natural person, but these deter- 
minations may not always be clear-cut. This scenario would also raise 
the question whether ‘natural person’ is still the most suitable term to 
be used for the purposes of the GDPR. 


e When DF technology is used to conduct live anonymous conversations 
where, for example, a person assumes the voice or face of a non-existent 
person, this will most likely fall outside the scope of the GDPR. However, 
even then, from a person’s choice of words or the information they share, 
it might be possible for someone to relate the communication back to 
them. In that case, personal data will be processed. 


e As the GDPR does not apply to deceased persons, in principle, a DF 
about a person who is no longer alive does not constitute the process- 
ing of personal data. However, often, the data may indirectly refer to 
a natural person, such as a relative. Again, this is a matter of scale. In 
general, the more sensitive the data, the more recently the person has 
died and the more the DF concerns or reflects the current state of affairs, 
the more likely the data are to be qualified as personal data. For exam- 
ple, if Cleopatra were brought back to life to simply say ‘Hey, what’s 
up?’, this would not qualify as indirect personal data with respect to 
her descendants. However, if Queen Elizabeth II were brought back 
to life to say ‘My son is a nitwit’, this would qualify as personal data 
relating to King Charles II. Most DFs of deceased persons will fall 
somewhere in between these two extremes and are consequently more 
difficult to categorise as either containing personal data or not. 
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¢ In principle, DFs about legal persons do not qualify as personal data. 
An example could be a DF in which the pumpjacks of Shell appear to 
spill most of the oil on the ground near a natural reserve, which leaks 
into a nearby river. However, if a company is run by one person, a DF 
about that company will generally qualify as personal data, because 
the conduct of a legal person is directly linked to the conduct of the 
natural person. Again, this is not a binary distinction, but a gradual 
one. The smaller the company, the more likely it is that a DF about 
a legal person will contain indirect personal data. In addition, if it is 
clear that the operations depicted fall under the responsibility of one 
specific employee, (eg, the operational manager of the facility where 
the pumpjacks of Shell appear to be leaking oil), this may qualify as 
personal data relating to that specific natural person. 


e DFs may also be used to merge the faces or voices of two or more 
persons. In general, the fewer persons of whom data are used, the 
more likely the DF is to hold personal data. In general, when the facial 
features of say 20 people or more are merged to create a new image, 
this will not contain personal data, yet even then, a person’s nose may 
be so unique that it might stand out and allow for identification. 


e DFs of non-existent persons will, in principle, not fall under the regu- 
latory scope of the GDPR, but it raises questions as to whether they 
perhaps should. In addition, it may often not be known to a subse- 
quent data controller with whom a DF is shared whether, for example, 
a photo is a depiction of an existing person or not. A scenario might 
arise in which a data controller states that they did not abide by the 
GDPR because they were convinced that the data they were process- 
ing related to a non-existent person. 


Figure 5.1 Non-existent persons generated by www.thispersondoesnotexist.com 
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e AR can be used to alter reality, such as the audio-visual feedback. 
In general, these AR devices will process both the personal data of 
the user (eg, through eye-tracking technology) and of people in the 
vicinity of the user. Yet when an AR device only records the shapes of 
objects the user encounters and, every time the shape of a human is 
detected, the object is shown as a cat or merely a silhouette, in prin- 
ciple, no personal data are processed. Similarly, the voices of others 
may be distorted. Still, even in this kind of scenario, specific persons 
may have unique body futures and in specific contexts, it may be clear 
who the person is even from the silhouette. 


e For VR, if an avatar directly resembles a physical person, data about 
the avatar will, in principle, qualify as personal data. If it is known 
who operates an avatar that does not resemble its user, data about 
the avatar will also qualify as indirect personal data. Similar to HRs. 
A complicated question might emerge concerning personhood and 
identity when people identify more with their (non-resemblant) 
avatar than with their own physical body or when they use the avatar 
to navigate life because they are mostly housebound. 


Table 5.1 The continuum of non-personal to personal data 


Non-personal data Personal data 

HR/avatar operated by HR/avatar operated by one person 
neighbourhood/large group 

Person does not identify with HR/ Person identifies more with HR/ 
avatar avatar than with physical body 
Natural person and HR distinct Natural person and HR merge 
Trivial DF about historical figure Sensitive DF about recently deceased 


person with surviving relatives 


General DF about large cooperation | Specific DF about one-man firm 


AR which replaces/blocks out AR which adds personal information 
humans to PR 


5.2.3. Data Quality 


The GDPR requires that data that are processed should be accurate 
and complete. The data quality provision finds its way back to the 1973 
Resolution on the processing of personal data in the private sector, 
adopted by the Council of Europe.’ The explanatory memorandum 
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provides that ‘computerised information can give a semblance of special 
reliability. Mistakes may cause serious damage, because of the intensive 
use that can be made of the data’. The 1974 Resolution on data process- 
ing in the public sector contained a similar rule, and the explanatory 
memorandum linked the need for the data quality principle to the fact 
that computerised information is usually regarded as being particularly 
accurate." 
Under the GDPR, the controller must ensure that personal data are: 


accurate and, where necessary, kept up to date; every reasonable step must be 
taken to ensure that personal data that are inaccurate, having regard to the 
purposes for which they are processed, are erased or rectified without delay.” 


The GDPR contains a correlating right: 


The data subject shall have the right to obtain from the controller without 
undue delay the rectification of inaccurate personal data concerning him or 
her. Taking into account the purposes of the processing, the data subject 
shall have the right to have incomplete personal data completed, including by 
means of providing a supplementary statement.® 


Importantly, when a data controller corrects information, updates it or 
complements incomplete data, the GDPR requires them to communicate 
that rectification: 


to each recipient to whom the personal data have been disclosed, unless this 
proves impossible or involves disproportionate effort. The controller shall 
inform the data subject about those recipients if the data subject requests it.’ 


Adhering to the requirement to inform all recipients of incorrect data 
may be complicated, especially when data are published online, as who 
has downloaded the incorrect information and further distributed it may 
not always be clear. Only when informing the other parties is dispro- 
portionate will this notification obligation be waived. What counts as 
disproportionate depends on the efforts that need to be taken by the data 
controller to reach out to those third parties, as well as on the potential 
consequences for the data subject if the incorrect personal data continue 
to be distributed. Thus, when it concerns a DF of non-consensual 
pornography that has been published online, the person who has created 
and uploaded the initial video must go to lengths to remove every copy 
from the internet. 

The data quality principle strikes at the heart of synthetic tech- 
nologies. By its very nature, Synthetic Reality (SR) is not accurate, as it 
twists or alters reality or its representation. Although it is unlikely that 
every manipulation, however small, will lead to a fine or penalty, the 
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GDPR does not contain a so-called ‘de minimis’ rule. This means that 
any processing of data, even the most mundane and factually correct 
data (such as writing on a blog ‘Angela Merkel has blue eyes’) will fall 
under the GDPR and that any violation of the GDPR, however small 
(for example, ‘Angela Merkel has brown eyes’) could lead to a sanction 
or penalty. There is no guidance on how the various ways in which the 
different synthetic technologies impact reality will be regulated under the 
data quality principle. 

The additional question is who has the final say on what is correct 
and what is not. Data subjects may, in some cases, have an interest in 
providing incorrect data and may favour certain interpretations of reality 
(eg, I am reliable) that are not supported by facts (eg, having defaulted on 
a loan before). The GDPR is silent on the question of where the burden 
of proof lies with respect to proving that personal data is correct or not, 
and what standard of proof should be applied. It will ultimately be for 
a Data Protection Authority or court to decide on such matters, but 
determining what is true or not may involve complex questions and may 
require data subjects to present facts about aspects of their private lives 
that they would rather not share. 

For HRs, the data quality principle raises complex questions, as it is 
primarily aimed at situations in which data represent reality (or attempt or 
suggest doing so), while HRs are a Physical Reality (PR). Potentially, if HRs 
are truly indistinguishable from humans, situations might emerge in which 
a HR pretends to be a human. An example could be where a HR poses as 
a student (a natural person) and takes the exam on their behalf. Although 
the data produced by the HR are not inaccurate in themselves — rather, 
the answers to the exams may be more accurate than if the natural person 
had taken the exam — the suggestion that the natural person has produced 
the data is inaccurate. These scenarios are, of course, already unfolding in 
LLMs that are used for generating essays and papers, such as ChatGPT. 

It should be stressed that if a person is shown in a better, unrealistic 
light using DF technology, this may also conflict with the data accuracy 
principle. Although no harm may be done to the data subject directly, 
others may be deceived and take different actions than they would other- 
wise have done. A basic example of this deceit may be a porn actor who 
fakes themselves engaging in certain requests from their fans or a politi- 
cian faking a huge crowd at their rallies, or potentially giving an address 
in a minority language they do not speak. The extent to which the data 
quality principle applies to such scenarios is also unclear. However, it 
is clear that the principles underpinning the GDPR should not only 
be upheld if the data subject invokes a right or if the data subject has 
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suffered harm; they are primarily duties of care that data controllers need 
to respect, irrespective of whether a data subject has filed a complaint or 
invoked their right. A DPA may, for example, impose a penalty on a data 
controller for violating the GDPR without a single data subject having 
invoked a right or issued a complaint. 

For AR, the representation of reality is not accurate. If a person is 
bleeding, but AR blocks the sight of blood from the AR user, per se this 
may affect the person’s view of reality and how they perceive others. But 
again, it is unclear to what extent this would conflict with the data accu- 
racy principle. More complicated is the question of adding information 
to reality using AR. If a person sees through their AR device that the 
person they are interacting with is active on behalf of a right-wing politi- 
cal party, they do not necessarily process any inaccurate information, but 
they do not see reality as others do either. Similarly, when an AR device 
allows a person to process infrared signals, this is not inaccurate, but it 
does alter the perception of reality. 

For VR, it is most likely that the data quality principle will not apply 
because it is not a representation of reality. As such, it is best compared 
to a film. However, a question may emerge when VR becomes so intuitive 
and so real that people start to believe VR to be PR and when both their 
brain and their body responds to it as such. A scenario could be envisaged 
where person A makes an avatar that closely resembles person B and lets 
that avatar do things person B would never do, while third parties, either 
consciously or unconsciously, attribute that behaviour to person B. 

Where the data quality principle does not apply, other areas of law 
may, such as criminal or civil law standards on identity theft, fraud 
and deceit. However, there are several limitations that come with this 
approach. First, both civil and criminal law usually require demonstrable 
harm to be applicable, which is not the case for the GDPR’s data qual- 
ity principle. Second, the data quality principle is obviously significantly 
broader than the specific doctrines in civil and criminal law. Third, while 
the GDPR is adopted at the EU level and is applicable far beyond the EU’s 
borders, civil and criminal law are not harmonised, so each European 
country has its own respective rules, which may differ from and even 
conflict with each other. 


5.2.4. Transparency 


The GDPR requires data processing to be transparent. This entails not 
only that the data subject has the right to request information from the 
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data controller about the who, what, why, where and how of the data 
processing operation, but also that the data controller has an independ- 
ent duty to actively inform the data subject of such information on its 
own initiative (independent of any request by a data subject). For this 
purpose, the GDPR distinguishes between two situations: first, the case 
in which the data were obtained from the data subject (ie, the controller 
was in direct contact with the data subject) — this is the case, for example, 
when a person uses an AR device through which the data of other people 
with whom they interact are recorded; and, second, the case where the 
data were ‘not obtained from the data subject’ — this is the case, for 
example, when a picture of a person is taken from the internet to gener- 
ate a DE" 


Table 5.2 Information duties of data controllers when data are and are not 
collected from the data subject directly 


Information to be provided where Information to be provided where 
personal data have been collected personal data have not been obtained 
from the data subject from the data subject 

— The identity and the contact de- — The identity and the contact de- 
tails of the controller. tails of the controller. 

— The purposes of the processing — The purposes of the processing 
for which the personal data are for which the personal data are 
intended, as well as the legal basis intended, as well as the legal basis 
for the processing. for the processing. 

— The recipients or categories of — The recipients or categories of 
recipients of the personal data, if recipients of the personal data, if 
any. any. 

— The period for which the personal | — The period for which the personal 
data will be stored. data will be stored. 

— The existence of the right to — The existence of the right to 
request from the controller access request from the controller access 
to and rectification or erasure of to and rectification or erasure of 
personal data or restriction of personal data or restriction of 
processing concerning the data processing concerning the data 
subject or to object to processing subject or to object to processing, 
as well as the right to data port- as well as the right to data port- 
ability. ability. 

— The existence of automated deci- |— The existence of automated deci- 
sion making, including profiling. sion making, including profiling. 

— From which source the personal 
data originate and, if applicable, 
whether it came from publicly ac- 
cessible sources. 
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When a HR or a person wearing an AR device interacts with people, 
each and every one of them must be informed in full of which data are 
being collected, as well as why, how and to what end. It is difficult to see 
how this could work in practice, as it would take at least a minute to 
provide all the relevant information. In addition, one of the underlying 
rationales of the duty to inform a data subject the moment that their 
data are collected is that they can either not consent to it or, if consent 
is not required (see section 5.2.5), invoke their right to object. Again, it 
is difficult to see how, in practice, arguments between the data controller 
and data subjects could take place. Moreover, the HR or person using an 
AR device must be able to differentiate between the people they interact 
with that consent to their data being processed and those who object. 
An additional complicating factor is that the provision of information 
should be tailored to the data subject in question. For people who have 
a fair understanding of AI and how HRs and AR devices process their 
data, the information should be different from when a HR or AR device 
is used in a retirement home, where there is generally less understand- 
ing of how these technologies function. Finally, when data are collected 
about a minor, both the right to consent and to object depend on paren- 
tal agreement. This includes situations in which a person wearing an AR 
device encounters a minor in the public domain. 

When a HR scans the internet for additional information about a 
person it encounters or an AR device projects information next to a 
person found on the internet (eg, their LinkedIn profile), this would fall 
into the second category of information. DFs can be based on mate- 
rial collected from a person directly, such as where a porn company 
acquires data from an actor to generate a DF to perform the most sexu- 
ally perverse requests from fans, or where a DF of an actor is used in 
a movie to perform dangerous stunts. In general, most DFs created by 
professional parties for professional use fall into the first information 
category and most DFs created by citizens, who take raw material from 
online sources, fall into the second information category. Still, when one 
person asks another person to take their photo in order for them to make 
a DF, this will fall into the first information category. 

When the second regime applies, even more information needs to 
be provided to the data subject, such as where data were retrieved from. 
However, the most important difference between the two regimes is the 
moment the information should be provided. While under the first regime, 
the information should be provided the moment the data processing 
begins, under the second regime, information should be provided within a 
reasonable period after obtaining the personal data, but at the latest within 
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one month. However, if the data are to be used for communication with the 
data subject, which is arguably the case when a person uses an AR device or 
when a HR interacts with a person, the information should be provided at 
the latest at the moment when the first communication to the data subject 
is made. Also, when a disclosure to another recipient is envisaged, the data 
subject should be informed no later than that disclosure. This will be the 
case when a DF is published online or shared with a large crowd. 


5.2.5. Purpose for Processing 


The GDPR provides that the controller must have a legitimate ground 
for processing ordinary personal data. It exhaustively lists six grounds: 
(1) consent by the data subject; (2) contractual agreement between the data 
subject and the data controller; (3) the data subject’s vital interests are at 
stake; (4) there is a legal basis for processing the data; (5) data process- 
ing is in the public interest; or (6) data processing is in the interests of 
the data controller and these interests override the interests of the data 
subject not to have their data processed." 

When public sector organisations use any of the four technologies, 
they must do so on the basis of a law assigning them competence to do so 
and process data only to the extent that doing so is in the public interest. 
In addition, they should assess to what extent data processing is neces- 
sary and proportionate in relation to the goal pursued and ensure that the 
subsidiarity requirement is respected, meaning that there are no other, less 
intrusive means available for the organisation to further the public interest. 
For example, because synthetic technologies generally involve the process- 
ing of substantial amounts of personal data, when non-technological 
ways of furthering the public interest exist, which do not involve gathering 
personal data but are equally effective, using a synthetic technology would 
violate both the necessity and the subsidiarity requirement. 

The necessity, proportionality and subsidiarity requirement also 
applies to private parties processing personal data. They can invoke 
any of three grounds: (1) consent of the data subject; (2) a contractual 
agreement between the data subject and the data controller; and (3) a 
legitimate interest of the data controller. The first two are based on the 
consent of the data subject, either directly for the processing of personal 
data or indirectly, agreeing to contractual obligations for which the other 
party needs to process their personal data: 


¢ The first scenario applies, inter alia, when a person consents to a HR 
gathering their data or to a DF being made of them. Consent, under 
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the GDPR, is a freely given, specific, informed and unambiguous 
indication of the data subject’s wish by which they, by a statement 
or by clear affirmative action, signify agreement to the processing of 
their personal data." The controller has to be able to demonstrate 
that the data subject has consented, and the data subject has the right 
to withdraw consent at any time.” 


e The second scenario applies when a person signs a contract in order 
to have a care HR in their home to provide for their basic needs, 
knowing it is necessary for the HR to process their personal data in 
the course of doing so. 


e The third and final relevant legitimate ground for private organisa- 
tions and citizens processing personal data is the legitimate interest 
of the data controller that transcends the interest of the data subject 
not to have their data processed.'* In some cases, the processing of 
personal data through synthetic technologies may be based on this 
ground. This may include situations in which HR care robots only 
temporarily store data about healthcare personnel entering a person’s 
home or when a nearly blind person uses an AR device to navigate the 
streets. In most cases, the assessment of which interest prevails may 
raise complicated questions. The number of data subjects involved 
should also be accounted for, as their combined interest not to have 
their data processed should be considered. 


If a private sector data controller processes personal data based on a legiti- 
mate interest or when a governmental organisation processes personal 
data in the public interest, the data subject has a right to object, mean- 
ing that they challenge the idea that the data processing is actually in the 
public interest or that the interest of the data controller should override 
their interests. Invoking a right does not mean that such requests should be 
granted per se. The controller might, for example, present their reasons for 
believing why they are authorised to process the data. The GDPR provides 
no in situ solution for resolving this conflict. Ultimately, it is for the Data 
Protection Authority or a judge to decide on the matter. Practically speak- 
ing, this means the data controller can process personal data until they are 
disallowed from doing so by a Data Protection Authority or a judge, but 
they risk substantial fines if they are found to be in violation of the GDPR. 
This means the current legal regime may give rise to legal uncertainty for 
both the data controller and the data subject. 

Importantly, if a synthetic technology is used to process sensitive 
personal data, the legitimate interest of the data controller overriding 
that of the data subject is not a legitimate processing ground. The GDPR 
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specifies that it is prohibited to process personal data revealing racial 
or ethnic origin, political opinions, religious or philosophical beliefs, or 
trade union membership, as well as to process genetic data, biometric 
data, data concerning health or data concerning a natural person’s sex 
life or sexual orientation. An exemption to this prohibition is either the 
explicit consent of the data subject, if a private party needs to process 
this kind of data (eg, a church processing the religious data of its 
members, a political party recording the convictions of constituents, an 
employer keeping a record of sick leave or a medical institution process- 
ing the health data of patients), when data processing is necessary for 
a substantial public interest or when a data subject has made such data 
manifestly public, such as when a politician gives a television interview 
to disclose that they are gay. "° 

Thus, potentially, when there is a medical reason to place a HR 
within a person’s home, processing the data of medical personnel may 
be legitimate under the GDPR, although this would need to be assessed 
on a case-by-case basis. When an AR device is used in a public setting, 
it is difficult to avoid processing sensitive data, such as when the indi- 
vidual wearing the AR device walks past two men kissing on a bench, 
a person in a wheelchair or a woman of colour wearing a headscarf. It 
would be for a judge or a Data Protection Authority to decide whether 
the ‘accidental’ or ‘unintended’ processing of data falls under the strict 
regime for processing sensitive data, but most likely it will. The GDPR 
places emphasis on the factual situation — are sensitive data processed? — 
not on the intended state of affairs. It should be stressed that wearing a 
headscarf is not considered as ‘manifestly making public’ one’s religious 
convictions, nor does this exemption apply to a person in a wheelchair 
going out in public or two men kissing on a bench. These are not aimed 
at disclosing sensitive information to the public. Because most DF porn 
is non-consensual and no other exemption to the prohibition to process 
sensitive data will apply, it is deemed to be in violation with this provi- 
sion in the GDPR, which could lead to a fine of up to €20 million." 

In addition to having a legitimate processing ground, the GDPR 
requires the data controller to establish a purpose for processing personal 
data before commencing. This purpose should be clear, unambiguous 
and specific. The purpose of collecting personal data to ‘enhance the 
functionality of the HR’ or to ‘provide for relevant services to the user’ 
is not specific enough; it needs to be clear why it is necessary to collect 
each data point for the specific purpose at hand, so the data processing 
can be evaluated in light of the principles of necessity, proportionality 
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and subsidiarity. The requirement of purpose specification may be diffi- 
cult to uphold with respect to synthetic technologies because it requires 
data controllers to formulate specific goals before gathering personal 
data and then assess which data can be gathered in light of that goal and 
which cannot in concrete contexts. However, some synthetic technologies 
simply process the data in their surroundings by default, without pre- 
determining or pre-selecting what data are relevant. 

In addition to the purpose specification principle, there is the purpose 
limitation principle. This principle entails that data collected for one 
specific purpose may, in principle, only be processed for that purpose 
and not for purposes that are incompatible with the original purpose for 
processing data.” DFs generated by professional parties (eg, the porn 
industry, the film industry and the retail sector) are based on material 
collected specifically for producing DFs. However, it is clear that most 
DFs generated by non-professional parties by far involve taking existing 
audio-visual material, usually from the internet, to create new audio- 
visual material. The original material is typically not published for the 
purpose of generating a DF nor would the purpose of the DF — for exam- 
ple, for satire or pornography — align with that original purpose. When 
AR devices or HRs find data on the internet and either project the data as 
a visual for the user or for take appropriate action vis-a-vis a person who 
is encountered, this may conflict with the purpose limitation principle as 
well. If the purpose for processing conflicts with the original purpose, 
such processing is prohibited, unless the data subject gives their consent. 

Finally, the GDPR contains a so-called ‘household exemption’. The 
GDPR does not apply when personal data are processed by a natural 
person in the course of a purely personal or household activity.'’ The 
Court of Justice of the European Union (CJEU) has issued two impor- 
tant judgments concerning this exemption. 

In the first case, a woman kept a personal blog on the internet on which 
she shared information and facts about acquaintances and colleagues, 
such as that one of them had broken a leg. The question emerged as to 
whether this information fell under the household exception, because the 
purpose for which the data were being processed was personal, and the 
internet page was mainly intended for the women herself and a small 
circle of acquaintances. The CJEU found otherwise, holding that: 


[The exception applies only to activities] which are carried out in the course 
of private or family life of individuals, which is clearly not the case with the 
processing of personal data consisting in publication on the internet so that 
those data are made accessible to an indefinite number of people.” 
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The second case concerned a person who had a camera aimed at the 
entrance to his property for safety reasons. Again, the Court took a 
restrictive approach: 


To the extent that video surveillance such as that at issue in the main proceed- 
ings covers, even partially, a public space and is accordingly directed outwards 
from the private setting of the person processing the data in that manner, it 
cannot be regarded as an activity which is a purely ‘personal or household’ 
activity.” 


In the context of synthetic technologies, three important questions flow 
from the foregoing examples: 


Where personal data are shared on the internet or with a large group 
of unselected people, the household exemption does not apply. When 
data are not shared with anyone or are only disclosed to one other 
person using a private channel, the exemption may apply. However, 
data may also be disclosed to a medium-sized group, which may be 
the case when a HR collects data about an elderly woman and the 
people who visit her, for example, and this data is only disclosed to 
her three children and eight grandchildren, or if a DF of a classmate 
is shared in a group of 10 friends. In such cases, the applicability of 
the household exemption may be more difficult to establish. As with 
the concept of personal data, the legal approach to the household 
exemption is a binary one, while a factual assessment depends on 
various incremental factors. 


If data are collected in the private sphere, the household exemption 
may apply. However, when data are gathered from the public domain, 
such as on the street or from the internet, it does not. Again, there 
will be grey areas, such as when a group of retired citizens acquire a 
HR that operates in their retirement home or when an AR device is 
used in the hallway of an apartment building. Should these be consid- 
ered semi-public or semi-private domains? 


In many cases, there will be multiple parties involved in processing 
the personal data, each for their own purposes, such as an AR device 
or HR owner and the company processing the data. The household 
exemption only applies to data processing by a natural person for 
private, non-commercial activities. It is unclear what the household 
exemption would mean if a natural person processed personal data 
for personal reasons, but used a technology provider to process the 
data on their behalf, and the latter had a commercial interest.” 
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The European data protection advisory organisation has urged the EU 
legislator to reconsider the household exemption in order ‘to reduce 
as far as possible the legal uncertainty that currently surrounds vari- 
ous aspects of individuals’ personal or household use of the interner.” 
However, thus far, no substantive legislative changes have been made, so 
legal uncertainty on this point remains. More fundamentally, the under- 
lying rationale for the exemption was that when people draw explicit 
pictures of their neighbours in their home, this does little harm, it 
would be impossible to enforce a legal provision that would prohibit this 
conduct and, if doing so were possible, this Orwellian scenario would be 
a cure that was worse than the disease. The second and third rationales 
still apply, but the harm-based rationale does not. DF pornography is 
far more realistic and persuasive than a drawing, and it is far easier for a 
person to make a full-length porn film than when doing so without digi- 
tal techniques. More significantly, the boundary between the private and 
the public domain is becoming increasingly fluid. The difference between 
DF pornography being consumed in private and being widely distributed 
on the internet is literally a mouse click. The question is consequently 
whether this exemption should still be upheld. 


Table 5.3 The continuum between the private and public spheres 


Household exemption does not apply 

Household exemption applies when: | when: 

— data are gathered in the private — data are gathered from the public 
sphere; and domain/internet; or 

— data are not shared or only toa — data are shared with large group 
small group; and or with anyone; or 

— data are shared via a private — data are shared via an open 
channel; and medium; or 

— for purely private or household — data are processed for purely 
activities; and commercial activities; or 

— no harm is inflicted. — substantial harm is inflicted. 


5.2.6. Conclusion 


This section has made three things clear. First, many of the current appli- 
cations and use cases of synthetic technologies simply conflict with data 
protection law. It is difficult to see how the transparency requirement can 
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be respected for most non-professional DFs, uses of AR devices and HRs 
that interact with a non-selected group of people. Equally, in the case of 
AR devices and HRs that gather all data about people they encounter at 
random, this would likely conflict with the purpose specification princi- 
ple. When data are re-used for different purposes, such as when online 
material is used for producing a DF or when an AR device projects data 
found on the internet next to the face of a person the user encounters, 
this may conflict with the purpose limitation principle. When synthetic 
technologies process sensitive personal data (eg, data regarding health, 
sex or race), this is in principle prohibited, unless an exception applies. 

Second, there are many instances where the application of the GDPR 
to synthetic technologies is not entirely clear. The GDPR is based on 
binary distinctions, such as between personal data and non-personal 
data, and between processing data for personal and non-personal activi- 
ties. However, in reality, these distinctions are much more granular and 
incremental. Thus, many of the legal determinations concerning the 
applicability of the GDPR and the interpretation of the requirements 
contained therein need to be performed on a case-by-case basis. This 
may raise a series of complicated questions for each and every instance. 
In addition, it is unclear how the data quality principle would and should 
apply to Al-generated realities. 

Third, on some points, the GDPR may need to be updated to be 
effective in a synthetic society. One question is, for example, to what 
extent the concept of personal data should be widened to encompass 
data relating to legal persons, deceased persons and potentially non- 
human entities. Another example that has not been discussed at length 
in this section, but is still important, is the fact that the GDPR contains 
a provision prohibiting automated decision making.” The provision is 
written in such narrow terms that it plays almost no role in practice.” 
It requires solely automated decision making, meaning that any human 
intervention may mean that the provision does not apply. Similarly, it 
requires legal effects or ‘similarly significant effects’, meaning that most 
effects fall outside the scope of the provision. The provision also lays 
down a range of exceptions to the prohibition. However, the underly- 
ing rationale to limit computerised decisions and require decisions to 
be made by humans is widely seen as important, especially when those 
decisions have a significant impact on citizens, such as for HRs that make 
decisions about the people they interact with. Thus, this provision may 
need to be updated in order to deliver on its goal. A final example is that 
the household exemption may need to be revised. The use of synthetic 
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technologies can do significant harm, even when used in private, given 
the ease with which content may be transferred from the private domain 
to an audience of thousands or even millions. 


5.3. FREEDOM AND PROPERTY 
5.3.1. Introduction 


This section homes in on freedom and property rights as acknowl- 
edged under the European Convention on Human Rights (ECHR) and 
the EU Charter of Fundamental Rights (CFREU). It explores what it 
would mean if non-human entities were granted legal standing under 
the ECHR, and to what extent natural persons have claim rights vis- 
a-vis synthetic technologies and entities (section 5.3.2). This section then 
focuses on the respect for mental integrity (section 5.3.3), prohibition of 
bias and discrimination (section 5.3.4) and finally, the conflict between 
the freedom of expression and the right to privacy (section 5.3.5). A brief 
conclusion is provided at the end (section 5.3.6). 


5.3.2. Rights for and Over Non-human Entities 


The discussion on a legal status and potential ‘human’ rights for non- 
human entities was briefly touched upon in section 4.5 in Chapter 4. In 
his book Robot Rights, David Gunkel discusses four potential positions 
on the question whether robots can and should have rights: 


(1) robots cannot have rights and should not have rights; 

(2) robots can have rights and should have rights; 

(3) although robots can have rights, they should not have rights; and 
(4) although robots should have rights, they cannot have rights. 


He concludes that none of these four positions is convincing.” If robots 
have a moral consciousness of sorts, they cannot be deprived of legal 
protection altogether, but at the same time, their position and conscious- 
ness are ultimately different from those of humans. Although it is difficult 
to imagine non-human entities having rights, the current legal regime does 
attribute human rights to legal persons, which are also entities that are 
created by natural persons. A company can, for example, invoke its right to 
protection of its ‘home’, as contained in Article 8 ECHR, when the police 
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enter its business premises.” In addition, ever more countries are attribut- 
ing rights to animals and even to natural phenomena such as rivers.” Thus, 
the attribution of human rights is far from limited to natural persons. 

The decision of whether to attribute rights to non-human entities 
is not only something that depends on the moral status of the entity 
in question. Conversely, the legal status of an entity has a considerable 
impact on its moral status. For centuries, women were afforded a lower 
moral status than men. This is why they were granted fewer rights and an 
inferior legal position. This inferior legal position also affected their soci- 
etal position and status, as well as their capacity to explore and develop 
in full. For non-human entities, the same may be true. Granting rights 
to non-humans would provide them with protection and it would allow 
them to flourish, develop and assert themselves in the world according to 
their own views. Not granting rights to non-human entities would mean 
that, essentially, they remain instruments to human ends.” 

The more dominant the role of non-human entities in society is and 
the more intelligent and sentient they become, the less counter-intuitive 
it will be to grant rights to them directly. Yet granting rights and legal 
persona to non-human entities would raise a range of difficult questions, 
such as the following: can non-human entities sue humans for being 
maltreated or should this be left to special advocates or interest groups? 
How should non-human entities substantiate harm, such as dignitary or 
emotional harm? Should non-human entities also have voting rights? Can 
non-human entities sue each other? Would these kinds of legal disputes 
be judged by human judges, non-human judges or a combined panel? 

Suppose that non-human entities were granted a legal persona 
and were allowed to invoke the human rights contained in the current 
catalogue. This would lead to several complicated questions. Some non- 
exhaustive illustrations are provided below: 


e Article 2 ECHR provides the right to life. No one shall be deprived 
of their life intentionally, save (1) in the execution of a sentence of a 
court, (2) in defence of any person from unlawful violence, (3) in order 
to effect a lawful arrest or to prevent the escape of a person lawfully 
detained, or (4) in action lawfully taken for the purpose of quelling 
a riot or insurrection. If this provision were applied to non-human 
entities, it would raise the question of what would amount to a non- 
human death. The difference between a human and a non-human life 
is that the latter is constructed. Although it can be deconstructed or 
destroyed, it can be put together again.” A robot that is switched off 
can be turned on again, a device that is demolished can be repaired 
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and an avatar that dies in a shooting game can be revived the next 
moment.” 


Article 3 ECHR provides the prohibition on torture and degrading 
treatment. There is a high barrier for this provision to apply, so it is 
unlikely that any discomfort to a non-human entity would amount 
to inhuman or degrading treatment. However, if a HR or avatar is 
employed for perverse sexual fantasies, it might. Similarly, should this 
provision apply to non-human entities, the generation of DF child 
pornography of non-existent children would likely be prohibited. 
At the same time, it is unclear to what extent these entities them- 
selves experience the treatment as degrading, especially when they are 
designed to perform and execute that exact function. 


Article 4 ECHR provides the prohibition on slavery and forced labour, 
although this does not include (1) any work required to be done 
over the ordinary course of detention, (2) any service of a military 
character or service exacted instead of compulsory military service, 
(3) any service exacted in case of an emergency or calamity threatening 
the life or wellbeing of the community, or (4) any work or service 
which forms part of normal civic obligations. If this provision were 
to apply to non-human entities, it would likely bring an end to the 
use of HRs for manufacturing or other jobs. It would also mean that 
they would have to decide for themselves which job they wanted to 
take and under which conditions. They should therefore be entitled 
to remuneration. Still, the use of HRs for warfare might be permitted 
as part of compulsory military service. 


Articles 5 and 6 ECHR specify rules on legitimate detention and the 
right to a fair trial, Article 7 contains the prohibition on retroac- 
tive legislation. This would entail that if non-human entities were 
incarcerated or brought before a court of law, the normal rules of 
procedure should apply (see section 5.5.2). In addition, it would not 
be allowed to retroactively punish non-human entities for conduct 
that took place at a time that such conduct was not prohibited. 


Articles 8, 9, 10, 11 and 12 ECHR respectively contain the right to 
privacy, freedom of religion, freedom of expression, freedom of 
association, and the right to marry and establish a family. It is clear 
that applying these to non-human entities would raise a number of 
complex questions. If non-human entities were to be given these 
rights, this would mean, inter alia, that the data of non-human enti- 
ties could not be processed without a legal basis or consent, and that 
they were free to express their own political views and to associate in 
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a union. Equally, they could not be forced to marry a person against 
their will. How the right to establish a family would work in terms of 
non-human entities is unclear. 


e Article 1 of the First Protocol to the ECHR grants the right to property. 
If this right were to be attributed to non-human entities, it would mean, 
inter alia, that these entities should be able to enjoy the fruits of their 
labour and would be entitled to possess objects. Article 2 of the Protocol 
provides the right to education, which would grant non-human enti- 
ties the right to develop themselves through education. Article 3 of the 
Protocol specifies the obligation for states to organise free and demo- 
cratic elections. Were this provision to be applied to non-human entities, 
it would mean that they would have the right to vote in elections. This 
would have an enormous impact on democratic elections, as in time, it 
is likely that there would be more non-human than human entities. In 
addition, political parties defending the rights of non-human entities 
could be founded and influence political decision making. 


e Finally, Article 14 ECHR contains the prohibition of discrimination. It 
is highly likely that if non-human entities were to be given legal stand- 
ing under the ECHR, the European Court of Human Rights (ECtHR) 
would find that a distinction between non-human and human entities 
could only be made if such a distinction were to be fair and reason- 
able in a specific context. It would put a limit on or outright prohibit a 
potential ban on interspecies marriages, voting rights for non-human 
entities and non-human entities entering buildings, for example. 


From the above, it is clear that complicated legal, moral and practical 
questions would emerge if the current human rights catalogue were 
to be applied to non-human entities. Potentially, a special non-human 
rights framework could be developed, in which case a distinction would 
be made between the rights of HRs, avatars, non-existent persons and 
deceased persons. It should be stressed that although entities may be 
attributed legal persona or rights, this does not necessarily mean that they 
have legal standing.*! Countries that have attributed rights to animals 
or natural phenomena obviously do not expect that non-human entity to 
go to court; instead, it allows natural or legal persons to go to court on 
their behalf. Another question that will need to be answered is the extent 
to which the Al-driven entity can be attributed an independent legal 
status if it is partially dependent on human input. If a person designs 
and creates their own avatar in VR, but that avatar then develops autono- 
mously through self-learning AI and is subsequently attributed rights, 
the avatar might make choices with which its creator disagrees. 
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Rights for 
non-human 
entities 


Figure 5.2 Different parameters that need to be taken into account when 
attributing rights to non-human entities 


Because it is clear that in the coming years, it is unlikely that rights will 
be attributed to non-human entities, the protection of and from non- 
human entities will come through humans invoking their rights under 
the existing human rights framework for the time being. A logical start- 
ing point looking at which provisions might be invoked is the right to 
physical and intellectual property, and the right to conduct a business, 
which is derived from those rights. 
Article 1 of the First Protocol to the ECHR states that: 


Every natural or legal person is entitled to the peaceful enjoyment of his 
possessions. No one shall be deprived of his possessions except in the public 
interest and subject to the conditions provided for by law and by the general 
principles of international law. The preceding provisions shall not, however, 
in any way impair the right of a State to enforce such laws as it deems neces- 
sary to control the use of property in accordance with the general interest or 
to secure the payment of taxes or other contributions or penalties. 


Article 17 CFREU provides: 


1. Everyone has the right to own, use, dispose of and bequeath his or 
her lawfully acquired possessions. No one may be deprived of his or her 
possessions, except in the public interest and in the cases and under the 
conditions provided for by law, subject to fair compensation being paid in 
good time for their loss. The use of property may be regulated by law in 
so far as is necessary for the general interest. 


2. Intellectual property shall be protected. 
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Finally, Article 16 CFREU states that: 


The freedom to conduct a business in accordance with Community law and 
national laws and practices is recognised. 


The right to property extends to both material and immaterial assets.** 
With respect to non-material assets, the ECtHR has considered whether 
the legal position in question gives rise to financial rights and interests 
and therefore has an economic value. It has thus accepted that intellec- 
tual property (eg, trade marks, copyrights and patents) and licences fall 
under the right to property. Current possessions and legitimate expecta- 
tions regarding future possessions may fall under the right to property. 
This kind of expectation must be made more concrete than a mere hope 
and be based on a legal provision or a legal act, such as a judicial deci- 
sion, bearing on the property in question. With respect to the right to 
intellectual property, the Court has accepted, inter alia, claims about the 
registration of a trade mark prior to the trade mark being registered, the 
right to publish a translation of a novel and the right to musical works 
along with the economic interests deriving from them, also by means of 
a licence agreement. The ECtHR has also accepted that Article 8 ECHR 
may also partially cover the right to intellectual property.” The right to 
property also includes portrait rights. 

With regard to the question of whether an interference with the right 
to property exists, the ECtHR adopts a broad approach. An interference 
includes, inter alia, revocation or change of conditions of licences affect- 
ing the running of businesses, introducing a state monopoly, rent control 
systems, loss of certain exclusive rights over land, the freezing of bank 
accounts and the imposition of positive obligations on landowners. It is 
legitimate for a government to limit or annul possession if this is in the 
general interest and has a legal basis. In these instances, the Court will 
assess whether a fair balance has been struck between the rights of the 
individual and the general interest. To this end, the ECtHR may assess 
whether the special circumstances of the case were sufficiently taken into 
consideration. Finally, the freedom to conduct a business is often charac- 
terised as a sub-right to the right to property. However, in the EU, it is an 
independent right recognised in the CFREU. The CJEU has emphasised the 
broad nature of this right, but has also accepted the need for restrictions.*4 

The right to property and intellectual property and the freedom to 
conduct a business may come into play in various relationships where 
synthetic technologies are used. Some of the most important points, 
questions and dilemmas raised by synthetic reality are highlighted below: 
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A citizen may claim to have a property right over their HR and, 
potentially, over their avatar, so that a significant limitation on the 
non-human entity may impair the exercise of the right to property. 
The ECtHR would need to establish the extent to which an avatar 
qualifies as immaterial property and/or as an object protected under 
the right to intellectual property. The latter may be the case for plat- 
forms in which the player is allowed to shape and develop their avatar 
according to their own views and ideas. 


Producers and providers of HRs and the platforms enabling VR may 
claim the right to property and freedom to conduct a business, inter 
alia, when making decisions that affect player avatars or HRs operat- 
ing in citizens’ homes. 

It is clear that conflicts may arise when two parties hold shared or two 
parallel property rights, for example, one party over a VR platform 
and another party over an avatar navigating that VR platform. 


If a HR or LLM is used to produce something (eg, write an essay for 
an essay competition, design a house or compose a concerto), the 
question might emerge as to who has the intellectual property right 
over it, if anyone. Is it the owner or user of the HR, the producer or 
provider of the HR, the HR, everyone or no one? Potentially, this 
would depend on the circumstances of the case and on the input 
received by the HR from the user, the input queues received from the 
context and the share of the pre-programmed algorithm in perform- 
ing the task. 


The extent to which a person whose HR is destroyed can claim 
damages for future possessions needs to be determined, such as if a 
HR produces physical or intellectual work. 


Citizens and professional parties that create content (eg, a DF, using 
synthetic technologies) may claim intellectual property rights over 
their content, while people whose information is used by synthetic 
technologies (eg, a DF of a person used in an advertisement) may 
claim that this violates their portrait right. If an AI generates a new 
song using the voice of an existing artist, two contrasting intel- 
lectual property rights might be invoked. How these issues can be 
resolved will depend on the circumstances of the case. 


Potentially even more problematic is that learning from specific 
sources or content producers, AI programs can generate work in 
the style of a certain artist. It can write a lyric in the style of Taylor 
Swift, produce a painting that Marlene Dumas could have produced 
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or make a joke that you could almost hear Eddie Izzard make. It is 
unclear who owns copyright over such Al-generated works. 


¢ The extent to which the portrait right of one person can be overridden by 
the right to parody by another needs to be established.” Similarly, there 
is limited jurisprudential guidance on how the fair use or private copy 
exemptions would apply to the creation of synthetic media. The extent 
to which using an image of a person for new content can be considered a 
‘citation’ is unclear, as is the extent to which the use of DF pornography 
for private consumption would fall under the private copy exception. 


e Finally, a more general question concerns the training data of AI, 
for which often the internet and other public recourses are scraped. 
The copyright of those texts, for example, those produced on blogs, 
websites and news sites, is often not respected by the companies 
developing AI products and services.” 


A second doctrine that may be invoked is the right to privacy. Article 8 
ECHR states: 


1. Everyone has the right to respect for his private and family life, his home 
and his correspondence. 


2. There shall be no interference by a public authority with the exercise of 
this right except such as is in accordance with the law and is necessary in a 
democratic society in the interests of national security, public safety or the 
economic wellbeing of the country, for the prevention of disorder or crime, 
for the protection of health or morals, or for the protection of the rights and 
freedoms of others. 


This doctrine is the broadest of all provisions under the ECHR and, argu- 
ably, the most important of all human rights.” Over time, the underlying 
rationale has moved from the obligation of a state not to abuse its power 
to individual and subjective rights of natural persons to protect their indi- 
vidual autonomy, human dignity and personal freedom. As such, the right 
to privacy has been reinterpreted by the ECtHR as a provision granting citi- 
zens the right to flourish and explore and develop their identity to the fullest. 

According to the ECtHR, states are under an obligation, inter alia, 
to allow individuals to receive the information necessary to know about 
and understand their childhood and early development because this is 
held to be of importance because of ‘its formative implications for 
one’s personality’.** Article 8 ECHR extents to the public and profes- 
sional sphere: 


[T]he concept of ‘private life’ is a broad term not susceptible to exhaustive 
definition. It covers the physical and psychological integrity of a person. It 
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can sometimes embrace aspects of an individual’s physical and social iden- 
tity. Elements such as, for example, gender identification, name and sexual 
orientation and sexual life fall within the personal sphere protected by 
Article 8. Article 8 also protects a right to personal development, and the 
right to establish and develop relationships with other human beings and the 
outside world.*? 


In essence, the Court has emphasised that Article 8 ECHR harbours 
a personality right, a right to fully explore and effectuate one’s iden- 
tity and personality, in private, social, professional and public settings. 
This reinterpretation of the right to privacy means that it is conceivable 
that harm to non-human entities that are generated and controlled by 
a natural person (partially or fully) may be said to cause indirect harm 
to a natural person.* The extent to which this is the case will need to 
be determined by the ECtHR, as well as the level of harm required to 
bring a claim under the scope of the ECHR. While the GDPR does not 
contain a de minimis rule, the ECHR does.*! This means that if mini- 
mal harm is inflicted, in principle this cannot lead to a successful claim 
under the ECHR. No exact boundary can be drawn between harm that 
meets this barrier and harm that does not. 

A second way in which Article 8 ECHR may come into play vis-a-vis 
synthetic technologies is in relation to the interests of deceased persons. 
Under the ECHR’s current mechanism, natural persons are allowed to 
raise complaints over the treatment of a deceased loved one if there is 
a sufficiently direct link and the natural person is indirectly harmed. In 
addition, a family member may, under certain circumstances, continue 
a claim that was submitted by a deceased loved one when that person 
was still alive. However, deceased persons do not have a subjective claim 
right and, more importantly, the protection of their interests on their 
own account (thus without it affecting a natural person) generally falls 
outside the protective scope of the ECHR. 

Although the question to what extent the protective scope on this 
point needs to be extended is decades old,” synthetic technologies 
make this matter particularly urgent. Some of the matters that may 
merit regulatory clarity include the following examples. Although 
some would not mind, many people would resent knowing that 
after their death, their partner might continue to speak to them 
or have them give a speech at their own funeral. Should it be up to 
every individual to decide what can and cannot be done with their 
body representation after death, or should the law set those stand- 
ards, either through absolute prohibitions or through default legal 
assumptions which can be overridden by individuals? Is it desirable 


118 Law 


and permissible to have deceased historical figures teach in schools? 
Is it desirable and permissible to have deceased artists give a tour ina 
museum? Is it desirable and permissible to feature deceased actors in 
films? Is it desirable and permissible to have a deceased person appear 
in a pornographic film? Is it desirable and permissible to let deceased 
artists give concerts? 


5.3.3. Mental Integrity 


A third way in which the right to privacy could play a role vis-a-vis 
synthetic technologies is as a protection from those technologies. For 
example, a person encountering a HR or a person wearing an AR 
device could invoke Article 8 ECHR. The right to privacy conceived 
as personality right under the Convention not only covers the process- 
ing of personal data, but also the so-called ‘chilling effect’ this may 
have: 


[This effect] describes a condition in which invasions of privacy or the fear 
thereof cause individuals to change their behaviors, abstaining from other- 
wise lawful activity not because of its proscription by society but out of 
fear of public association with that activity ... The concept of ‘Chilling 
Effects’ is defined by two characteristics: (1) that individuals perceive there 
is at least a risk of their activity or condition being observed by another who 
may disseminate those observations; and (2) that individuals change their 
behavior-discontinuing or hiding that activity or condition-out of fear of 
public association with that activity or condition.¥ 


A good example of a judgment in which the ECtHR has accepted 
an application by a claimant based on this effect is a case in which 
the applicant complained that the designation of a security risk area 
by the Burgomaster of Amsterdam violated his right to respect for 
privacy, as the Burgomaster enabled a public prosecutor to conduct 
random searches of people over an extensive period in a large area 
without this mandate being subject to any judicial review. The govern- 
ment, on the contrary, argued that the designation of a security risk 
area or the issuing of a stop-and-search order had not in itself consti- 
tuted an interference with the applicant’s private life or liberty of 
movement. Since the event that was complained of, several preventive 
search operations had been conducted; the applicant had not been 
subjected to further attempted searches. According to the government, 
this was enough to show that the likelihood of interference with the 
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applicant’s rights was so minimal that this deprived him of the status 
of victim. 

The Court stressed that, in principle, it did not accept in abstracto 
claims or an actio popularis. This refers to claims in which an applicant 
does not claim to have suffered harm themselves, but claims that a law or 
public policy is, as such, in violation of the rule of law or in conflict with 
the general interest: 


In principle, it is not sufficient for individual applicants to claim that the 
mere existence of the legislation violates their rights under the Convention; 
it is necessary that the law should have been applied to their detriment. 
Nevertheless, Article 34 entitles individuals to contend that legislation 
violates their rights by itself, in the absence of an individual measure of 
implementation, if they run the risk of being directly affected by it; that is, 
if they are required either to modify their conduct or risk being prosecuted, 
or if they are members of a class of people who risk being directly affected 
by the legislation. 


The Court went on to conclude that in this case, the applicant satisfied 
the victim requirement: 


The Court is not disposed to doubt that the applicant was engaged in lawful 
pursuits for which she might reasonably wish to visit the part of Amsterdam 
city centre designated as a security risk area. This made her liable to be 
subjected to search orders should these happen to coincide with her visits 
there. The events of 19 February 2004, followed by the criminal prosecution 
occasioned by the applicant’s refusal to submit to a search, leave no room for 
doubt on this point. It follows that the applicant can claim to be a ‘victim’ 
within the meaning of Article 34 of the Convention and the Government’s 
alternative preliminary objection must be rejected also.* 


Thus, not only when personal data are in fact processed but also when 
a citizen amends their behaviour, either by restricting their activities to 
appear ‘normal’ when they know they are or may be monitored or when 
they avoid areas in order not to be monitored, such may result in inter- 
ference under the terms of Article 8 ECHR. Citizens might rely on this 
chilling effect, for example, when a person amends their behaviour or 
avoids the places in which they think they might be confronted with a 
HR or AR device that is likely to process their data. Also, when the HR 
or AR device in fact does not process their personal data, but the person 
thinks it might, there could be a chilling effect. This is consequently an 
important addition to the right to data protection (see section 5.2), which 
depends on a factual, not a presumed state of affairs. It should be under- 
lined that an interference with the right to privacy might be justified if it 
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is based on a law, serves a public interest and is necessary in a democratic 
society. The extent to which this is true is determined on a case-by-case 
basis by the ECtHR. 

An important addition is the right to mental integrity, which the 
ECtHR derives from the right to autonomy it reads into the right to 
privacy. The Court has found that notions of ‘personal autonomy and 
quality of life’ underpin Article 8 ECHR, for example in the medical 
sphere,** and that the provision also protects a person’s mental, psycho- 
logical and moral integrity.” 


Private life, in the Court’s view, includes a person’s physical and mental 
integrity. The guarantee afforded by Article 8 of the Convention is primar- 
ily intended to ensure the development, without outside interference, 
of the personality of each individual in his relations with other human 
beings.*8 


For the Court, the integrity of the body and the mind are among the 
strongest rights distilled from the right to privacy, which, in principle, 
cannot be remedied by invoking paragraph 2 of Article 8 ECHR, save in 
exceptional circumstances. It is true that, so far, the Court’s main focus 
with respect to mental, psychological and moral integrity have been 
cases in which physical harm was also done, or bodily integrity played 
a role, such as in cases of rape, degrading treatment or the refusal for 
sex-reassignment surgery. Still, over the years, the ECtHR has attributed 
a wider and more prominent role to mental integrity so that it cannot 
be excluded that such a right may be extrapolated to the context of 
synthetic technologies. 

This could be important because AI technologies can and are used to 
manipulate people. A plain and innocent example is when an AR device 
nudges a person to go to a shop on their way home. If this is successful, a 
manipulated person no longer identifies with their previous preferences. 
Although, currently, most forms of manipulation are not so success- 
ful that people fully identify with the new set of preferences, the more 
powerful AI technologies become in reshaping people’s worlds and their 
perception of it, the more likely it will be that people will identify in full 
with the new reality. 

This kind of manipulation can work in several ways. Manipulation 
can occur through altering choice design or removing certain options 
altogether. In particular, AI technology is used for ‘nudging’, which 
leaves all choice options open to people, but subconsciously steers them 
in certain directions:? 
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Nudges influences choice behavior in a variety of ways that include: 1) provi- 
sion of information (e.g. leaflets about the benefits of climbing stairs), 2) 
correcting misapprehensions about social norms (e.g. informing individuals 
of peer group behavior such as statistics of average alcohol intake), 3) alter- 
ing the profiles of different choices (eg, making healthy food appear more 
prominent in the canteen), 4) implementing default options (eg, changing 
an organ donation legislative system from opt-in to opt-out). The guiding 
principle behind these examples is to make the ‘better’ option more conveni- 
ent or salient for the decision-maker to select; this option is better because it 
maximizes future health, wealth, and wellbeing.” 


These examples are mostly pre-digital. As Karen Yeung explains, the use 
of modern technologies makes those nudges incomparably stronger and 
more effective: 


Big Data driven nudging is therefore nimble, unobtrusive and highly potent, 
providing the data subject with a highly personalised choice environment — 
hence I refer to these techniques as ‘hypernudge’. Hypernudging relies 
on highlighting algorithmically determined correlations between data 
items within data sets that would not otherwise be observable through 
human cognition alone (or even with standard computing support. 
thereby conferring ‘salience’ on the highlighted data patterns, operating 
through the technique of ‘priming’, dynamically configuring the user’s 
informational choice context in ways intentionally designed to influence 
her decisions.>! 


Susser et al suggest that these hypernudges are manipulative actions par 
excellence: 


Understanding manipulation as hidden influence helps to distinguish it from 
other forms of influence. In what follows, we distinguish it first from persua- 
sion and coercion, and then from deception and nudging. Persuasion — in the 
sense of rational persuasion — means attempting to influence someone by 
offering reasons they can think about and evaluate. Coercion means influ- 
encing someone by constraining their options, such that their only rational 
course of action is the one the coercer intends. Persuasion and coercion carry 
very different, indeed nearly opposite, normative connotations: persuading 
someone to do something is almost always acceptable, while coercing them 
almost always is not. Yet persuasion and coercion are alike in that they are 
both forthright forms of influence. When someone is trying to persuade us or 
trying to coerce us we usually know it. Manipulation, by contrast, is hidden — 
we only learn that someone was trying to steer our decision-making after the 
fact, if we ever find out at all.’ 


As the doctrine currently functions it is unclear to what extent the protec- 
tion of mental, psychological and moral integrity under the jurisprudence 
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of the ECtHR is sufficient to address the concerns surrounding mental 
integrity and autonomy with respect to synthetic technologies.” The 
same applies to several notions contained in the EU legal acquis, includ- 
ing, but not limited to, the following: 


The CFREU:** 


o Everyone has the right to respect for their physical and mental 


integrity. 


The AI Act:°° 


o Aside from the many beneficial uses of AI, it can also be misused 


and provide novel and powerful tools for manipulative, exploita- 
tive and social control practices. These practices are particularly 
harmful and should be prohibited because they contradict EU 
values of respect for human dignity, freedom, equality, democracy 
and the rule of law and EU fundamental rights, including the right 
to non-discrimination, data protection and privacy, and the rights 
of children. 

The following AI practices shall be prohibited: placing on the 
market, putting into service or the use of an Al system that deploys 
subliminal techniques beyond a person’s consciousness in order to 
materially distort a person’s behaviour in a manner that causes or 
is likely to cause that person or another person physical or psycho- 
logical harm; the placing on the market, putting into service or use 
of an AI system that exploits any of the vulnerabilities of a specific 
group of persons due to their age, physical or mental disability, in 
order to materially distort the behaviour of a person pertaining to 
that group in a manner that causes or is likely to cause that person 
or another person physical or psychological harm. 


The Unfair Commercial Practices Directive:*° 


o To materially distort the economic behaviour of consumers’ 


means using a commercial practice to appreciably impair the 

consumer’s ability to make an informed decision, thereby causing 

the consumer to take a transactional decision that he would not 

have taken otherwise. 

A commercial practice shall be unfair if: 

(a) it is contrary to the requirements of professional diligence; 
and 

(b) it materially distorts or is likely to materially distort the 
economic behaviour with regard to the product of the average 
consumer whom it reaches or to whom it is addressed, or of 
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the average member of the group when a commercial practice 
is directed to a particular group of consumers. 


Commercial practices which are likely to materially distort the 
economic behaviour only of a clearly identifiable group of consum- 
ers who are particularly vulnerable to the practice or the underlying 
product because of their mental or physical infirmity, age or 
credulity in a way which the trader could reasonably be expected 
to foresee, shall be assessed from the perspective of the average 
member of that group. This is without prejudice to the common 
and legitimate advertising practice of making exaggerated state- 
ments or statements which are not meant to be taken literally. 


¢ The Digital Services Act:°” 


o Providers of online platforms shall not design, organise or operate 
their online interfaces in a way that deceives or manipulates the 
recipients of their service or in a way that otherwise materially 
distorts or impairs the ability of the recipients of their service to 
make free and informed decisions. 


5.3.4. Bias and Discrimination 


Article 14 ECHR holds: 


The enjoyment of the rights and freedoms set forth in this Convention shall 
be secured without discrimination on any ground such as sex, race, colour, 
language, religion, political or other opinion, national or social origin, asso- 
ciation with a national minority, property, birth or other status. 


There are several reasons why the provision is only partially relevant to 
the context of synthetic technologies. 

First, the ECtHR differentiates between direct discrimination (deci- 
sions taken on the basis of one of the factors listed in Article 14 ECHR) 
and indirect discrimination (decisions not taken on the basis of one of 
those factors, but that have a substantial negative impact on groups 
with a certain racial, political or sexual background). There are only 
a few procedural safeguards the Court has laid down with respect to 
discriminatory practices. This has clear effects for the applicability of 
non-discrimination law in the AI context. Most codes and frameworks 
on ethical AI focus on the process leading up to a decision, such as which 
data are collected, how are they categorised, which algorithm is deployed, 
how correlations are interpreted, how diverse the group operating the AI 
system is and so forth. However, these matters do not fall within the 
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scope of non-discrimination law, which focuses not on the process lead- 
ing up to a decision, but on the decision itself and the effects thereof. 

Second, the fact that the ECtHR will only deal with submissions if 
citizens can successfully demonstrate that they have suffered harm means 
that it is difficult to address systemic and structural racism, as the effects 
are generally not individualisable. The aggregate or cumulative effect of 
AI systems may also be difficult to handle under the non-discrimination 
regime, as each of the individual infringements may have an insuf- 
ficiently harmful effect to bring it under the scope of the Convention, 
while the cumulative impact may be substantial. The ECtHR’s case-by- 
case approach, which assesses the harm done to a particular person in 
a particular context, means that the Court will only issue decisions on 
what needs to be done in order to remedy a harm in a particular case. As 
a consequence, some parties pay the damages or penalties in a specific 
case, but leave the underlying system, policy or technology that caused 
the infringement intact.’ Under the current regulatory paradigm, it is up 
to an individual to recognise harm and prove that such is the result of a 
discriminatory practice, while this may be far from clear given the ubiq- 
uity and obliquity of some of the synthetic technologies and their effects. 

Third, Article 14 ECHR has an ancillary status, meaning that citi- 
zens cannot complain under the Convention of a discriminatory practice 
or policy as such, but only if the discriminatory practice or policy has 
affected one of the rights contained in the Convention, such as the right to 
privacy, the freedom of expression or the right to a fair trial. This further 
limits the relevance of the anti-discrimination provision contained in the 
ECHR as it is unclear with respect to many of these doctrines how they 
apply to synthetic technologies, if at all. Thus, even if it was clear that 
synthetic technologies discriminate on the basis of one of the grounds 
contained in Article 14 ECHR, if the discriminatory practice does not 
affect one of the other rights contained in the Convention, a citizen’s 
application on this point will not be declared admissible. 

Fourth, Article 14 ECHR only contains a limited number of discrim- 
inatory grounds. Although there is a ‘catch-all’ provision, referring to 
‘other status’, the ECtHR has only been prepared to accept grounds under 
this category that are directly related to the grounds explicitly mentioned 
in the provision, such as gender identity, sexual orientation, health and 
disability, parental and marital status, and immigration status. It is 
unclear what this provision entails when explicitly discriminatory poli- 
cies are based on other factors, such as the type of smartphone a person 
has, the colour of their hair or their shoe size. AI systems have biases and 
may use a wide range of factors, including factors that are unrelated to 
grounds on the basis of which humans have traditionally discriminated. 
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Fifth, anti-discrimination law has not only been developed to prevent 
the type of historical discrimination; it is also based on the belief that 
the number of factors on which a policy or decision is based are rela- 
tively small and that these factors remain relatively stable. This results 
from the human limitations in terms of mental capacity and the practical 
investments in time, energy and resources required to change and update 
the criteria for policy and decision-making processes. AI poses new chal- 
lenges to the standard approach to anti-discrimination law. Policies and 
decisions can, in principle, be taken not on five relevant factors, includ- 
ing race and sexual orientation, but on 5,000 of them. Clearly, when one 
or more of the five factors is a discriminatory ground, this may raise a 
red flag, but the larger the number of factors and the smaller the rela- 
tive weight of the discriminatory ground, the more difficult this is to 
establish. 

Sixth, in AI systems, the factors used to make decisions may change 
at any point in time. This makes it very difficult to verify in hindsight 
what factors led to a decision. 

Seventh, with Big Data and AI, it is possible to assess the effects of 
policies and decisions on groups in detail. This may raise difficult ques- 
tions as any data point is indirectly correlated to a discrimination ground. 
The question is to what extent all these indirect correlations should be 
taken into account. In addition, every policy will have different effects on 
different groups of people. A policy to grant tax benefits to people living 
in rural areas may, for example, have the effect that white males are the 
main beneficiary, because immigrants and single women tend to relocate 
to municipal areas. A city’s policy to subsidise cultural institutes may be 
of more benefit to women than man, because the latter group, on aver- 
age, frequents museums, concert halls and theatres less often. All policies 
will have indirect discriminatory effects on all groups; again, the more 
data become available about these effects on different groups, the more 
urgent the question is of where to draw the line. 

Eighth, perhaps counter-intuitively, the ECHR’s main concern is not 
that of laying down human rights, but laying down principles concern- 
ing the rule of law. States can and need to enter a citizen’s home, set 
restrictions on freedom of speech, and limit the freedom of citizens in 
general. What the ECHR requires is that if states interfere with human 
rights, they do so in a non-arbitrary way. This also holds true for the 
non-discrimination provision. The ECHR does not prohibit states from 
differentiating between people on the basis of colour, religion or gender, 
but it requires this differentiation to be relevant. In some cases, differen- 
tiation, even on the grounds mentioned in Article 14 ECHR, may even be 
required. If the police only enter the homes of Mormons because they have 
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received reliable intelligence that among the small Mormon community in 
a city, several terrorist attacks are prepared, this may be deemed legitimate. 
The reverse is also true: if the police have information that a male Muslim 
between the ages of 20 and 30 is going to commit a terrorist attack at a 
certain train station, not only can they use this information as relevant 
criteria, but they should do so. If the police were to perform body cavity 
checks on everyone present at the train station, including white females 
over 60, this might be deemed a human rights violation because there would 
be no good ground for doing so. Because the state should keep human 
rights interferences to a minimum, it should operate in the most effective 
way: it should discriminate. In the AI context, this means three things: 


e The core of anti-discrimination law is to assess whether there were 
good reasons to discriminate; black box systems in general will not 
be deemed acceptable. If an organisation deploying an AI system 
cannot explain why it was necessary and reasonable to differenti- 
ate between relevant groups, the ECtHR will find a violation on this 
point. The burden of proof is on the claimant/victim to prove that 
they suffered harm from a discriminatory policy or decision, but the 
burden of proof is on the organisation making the decision or policy 
to demonstrate that such differentiation was objectively reasonable. 


e Non-black box AI may make it easier for organisations to demon- 
strate an objectively reasonable ground to differentiate because it can 
almost always point to a correlation between one data point (a proxy) 
and the data point that is relevant for its decision or policy goal. 


e However, this may also mean that it should be possible to find alter- 
native proxies that are not directly or indirectly discriminatory more 
easily. Here, courts may need to lay down a standard because finding 
alternative proxies may come at the price of effectiveness. Suppose 
the police department in City A operates HR patrol agents on the 
basis of predictive policing. Clearly, the police department has no 
data on who will commit a crime, so it uses proxies. It has several 
data points that may help, which have different predictive values. 
Suppose the highest predictor is the combination of age, gender and 
ethnicity, as 80 per cent of all crimes in City A are committed by 
non-white males aged 15-35. The police can also use location-based 
predictions, such as postcodes, which have a 70 per cent predictive 
value for where crimes will take place. However, postcodes have an 
indirect correlation with ethnicity. If police activities are based on 
postcodes, it would mean that 80 per cent of all people subjected to 
surveillance activities would be non-white (both male and female). 
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Alternatively, the policy could be based on time slots: most crime in 
this situation takes place between 2 am and 4 am. However, this only 
has a predictive value of 60 per cent and would still have an indirect 
effect on people of colour, because they are overrepresented in the 
group of people with a night jobs. 


Ninth, the ECtHR grants a rather wide margin of appreciation to coun- 
tries to set legitimate policy goals, even though they may implicitly or 
explicitly favour certain groups. For example, countries are permitted 
to disallow same-sex marriages, they can give advantages to married 
couples in their tax system, they are allowed to incentivise certain social 
and moral norms being thought at school, and it is permitted to favour 
certain religious organisations over others. Although there are limits on 
this margin of appreciation, countries can adopt differential policies if 
this is the outcome of a legitimate democratic process. This is a way 
in which the democratic process may keep intact or deepen existing 
inequalities. 

Tenth and finally, AI often has an additional self-reinforcing effect. 
For example, a predictive policing system may suggest that the police 
should monitor a certain neighbourhood (with a significant immigrant 
population) more than others, resulting in more data about crimes in that 
neighbourhood (and its inhabitants). This would lead to the AI system 
recommending even stronger monitoring of that neighbourhood. In a way, 
this is precisely what anti-discrimination law mandates. It requires good 
grounds for the police to monitor a certain neighbourhood, and a higher 
crime rate in that area may be exactly that. Courts generally do not require 
police units to assess the origins of social inequalities, let alone correct 
them. This is important because although many ethical frameworks for 
Al consider potential corrections for historical errors, non-discrimination 
law does not necessarily do so. This is problematic because AI is almost 
without exception trained on hugely biased data, whether they are scraped 
from the internet, drawn from law enforcement databases or taken from 
medical data pools, if only because drugs have traditionally almost exclu- 
sively been tested on white males between the ages of 20 and 40. 


5.3.5. Freedom of Expression 


Article 10 ECHR holds: 


1. Everyone has the right to freedom of expression. This right shall include 
freedom to hold opinions and to receive and impart information and ideas 
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without interference by public authority and regardless of frontiers. This 
Article shall not prevent States from requiring the licensing of broadcast- 
ing, television or cinema enterprises. 


2. The exercise of these freedoms, since it carries with it duties and respon- 
sibilities, may be subject to such formalities, conditions, restrictions or 
penalties as are prescribed by law and are necessary in a democratic society, 
in the interests of national security, territorial integrity or public safety, for 
the prevention of disorder or crime, for the protection of health or morals, 
for the protection of the reputation or rights of others, for preventing the 
disclosure of information received in confidence, or for maintaining the 
authority and impartiality of the judiciary. 


As with the right to privacy, the ECtHR has granted a broad scope to the 
freedom of expression. Five points are important in light of synthetic 
technologies: 


First, the freedom of expression includes not only the right to 
disseminate and receive factual information, but also subjective qual- 
ifications and opinions.” 


Second, the law includes the freedom to engage in satire, although the 
Court sets limits.’ The Court has even accepted that the publication 
of a fictitious news article could be legitimate under the Convention: 


In this article the first applicant had wished to criticise the national hysteria 
after Mr Maier’s accident. The essential statement behind the impugned ficti- 
tious quotation of Mr Eberharter’s thought was that he had every reason to 
be happy about his strong rival dropping out and the consequential chance 
of his winning, but had not expressed this openly. In reality, Mr Eberharter 
had had extraordinary ski-racing successes after Mr Maier’s injury. Almost 
everyone in Mr Eberharter’s position would have had the same thought deep 
down inside and the statement did not imply that he had reprehensible char- 
acter traits. In any event, it was clearly recognisable that he had not expressed 
such words at all.6! 


Third, the right to freedom of expression includes the right to make 
controversial and offensive statements: 


The Court’s supervisory functions oblige it to pay the utmost attention to 
the principles characterising a ‘democratic society’. Freedom of expression 
constitutes one of the essential foundations of such a society, one of the basic 
conditions for its progress and for the development of every man. Subject to 
paragraph 2 of Article 10, it is applicable not only to ‘information’ or ‘ideas’ 
that are favourably received or regarded as inoffensive or as a matter of 
indifference, but also to those that offend, shock or disturb the State or any 
sector of the population. Such are the demands of that pluralism, tolerance 
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and broadmindedness without which there is no ‘democratic society’. This 
means, amongst other things, that every ‘formality’, ‘condition’, ‘restriction’ 
or ‘penalty’ imposed in this sphere must be proportionate to the legitimate 
aim pursued. 

e Fourth, the Court has a standard approach with regard to cases 
where one party invokes the right to privacy (including reputational 
interests) and the other party invokes the freedom of expression. The 
criteria it considers when it comes to a publication which may inter- 
fere with the right to privacy of a person are: 


o the contribution of the publication to a debate of public interest; 

o the degree to which the person concerned is well known; 

o the prior conduct of the person concerned; 

o the method of obtaining the information and its veracity 
concerned; and 

o the content, form and consequences of the impugned publication. 


When it comes to a potential breach of a person’s reputation, the 
relevant criteria include: 


o the existence of an objective link between the impugned statement 
and the person; 

o the level of seriousness of the attack on the person’s reputation; 

o the benchmarks and elements in assessing whether the interference 
was proportionate to the legitimate aim of the protection of repu- 
tation, which include: 

o substantive elements, such as the form and manner of expres- 
sion and the distinction between factual statements and value 
judgements, and 

o contextual elements, such as the function and familiarity of the 
person. 


e Fifth, the ECtHR has handed down a good number of cases on 
freedom of expression on the internet. Among other things, it has 
recognised that the internet is a breeding ground for creative expres- 
sion, such as through memes and other funny and artistic expressions, 
that anonymity can be an important part of the right to freedom 
of expression, especially in countries where the government severely 
sanctions political criticism, and that, broadly speaking, the same 
standards apply to making expressions online as to making them 
offline. 


Of the cases in which a conflict between the right to privacy and the right 
to freedom of expression emerges, two are of special importance. 
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First, it is generally accepted by the Court that where a person 
discloses private information themself or where data are gathered from 
the professional or public domain, a person’s right to privacy is more 
limited, yet not non-existent. This situation may arise where a person has 
made information public themselves, which is later re-used (for example, 
in a DF) when data that are re-used for a DF have been gathered from the 
professional or public domain, or when an AR device or a HR is used in 
a public or professional setting. 

Second, the legitimate claim of public figures to privacy is gener- 
ally more limited than with respect to non-public figures, yet again, is 
not non-existent. This situation may arise where a DF is made about a 
public figure, such as a political leader, a member of a royal family ora 
celebrity, or when an avatar resembles a public figure. 

In both scenarios, it is relevant to discuss the approach taken by 
the ECtHR. In the first scenario, this is the ‘reasonable expectation of 
privacy’ doctrine, while in the second scenario, the ‘legitimate expec- 
tation of privacy’ doctrine applies. Both doctrines are outlined below. 
Given the ECtHR’s case-by-case approach, no general framework exists 
that can be discussed; instead, several important cases and the Court’s 
judgement in those matters are provided by way of illustration. 


Reasonable Expectation of Privacy 


e The first time the doctrine of ‘reasonable expectation’ was used 
was in a case in which the applicant alleged that, as a result of 
her complaints of gender discrimination, she had been subjected to 
surveillance, including eavesdropping on her office and intercept- 
ing conversations on her private home phone and her office phones. 
The applicant had two phones in her office: one with an exter- 
nal number for personal calls and one for police work. The calls 
on both these phones were paid for by the police, her employer. 
Consequently, the government argued that ‘the applicant had no 
reasonable expectation of privacy in relation to those telephones’.® 
However, the Court emphasised that the business premises can, in 
certain circumstances, constitute someone’s ‘home’ as protected 
under Article 8(1) ECHR, that ‘private life’ can also take place 
at one’s workplace and that ‘correspondence’ includes communi- 
cations from both private and business phones. Thus, the Court 
found that the applicant had a reasonable expectation of privacy 
vis-a-vis her telephone calls. 
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In a subsequent case, the Court went a step further when the appli- 
cants complained that listening devices had been used while they 
were at the police station. It stressed that people have privacy even in 
public spaces, especially when systematic or permanent records are 
made.” 


When another applicant’s telephone, email and internet use were 
monitored by the employer, the ECtHR found that: 
[The applicant] had been given no warning that her calls would be liable 
to monitoring, therefore she had a reasonable expectation as to the 
privacy of calls made from her work telephone. The same expectation 
should apply in relation to the applicant’s e-mail and Internet usage.°* 


A university made recordings of teachers in classrooms, possibly to 
assess their teaching quality. The ECtHR found that although they are 
not private places, classrooms are workplaces where social relation- 
ships are developed. Thus the lecturers had a reasonable expectation 
of privacy in their classroom.” 


A remarkable step was taken in a case in which the applicant had 
joined the Lancashire Probation Service and was involved in the 
treatment of sex offenders. He was also a director of Roissy, an 
organisation that advertised on the internet as a builder and supplier 
of BDSM products and an organiser of BDSM events and perfor- 
mances. A photograph of the applicant wearing a mask with two 
half-naked women was in circulation. Roissy was registered at the 
applicant’s address and the website contained links to a number 
of BDSM websites advertising various events and featuring photo- 
graphs of the applicant and others, half-naked, while performing acts 
that, according to the accompanying text, had taken place at a local 
private members’ club and involved male domination over submis- 
sive women. He was dismissed from his job because his conduct was 
deemed incompatible with his treatment of sex offenders. The Court 
recognised that the nature of the acts was apparent from the inter- 
net photos and advertisements. It acknowledged that his conduct and 
openness about it: 
could give rise to doubts as to whether the applicant’s activities may be 
said to fall with the scope of private life and, if so, whether ... there has 
been a waiver or forfeiture of the rights guaranteed by Article 8. The 
Court notes, however, that the applicant’s performances took place in 
a nightclub which was likely to be frequented only by a self-selecting 
group of like-minded people and that the photographs of his act which 
were published on the internet were anonymised.” 
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Therefore, the ECtHR was prepared to assess the case on its merits 
on the assumption that the applicant had a reasonable expectation 
of privacy even in this case. 


A final important step was taken when an applicant’s internet connec- 
tion was monitored because he was distributing child pornography. 
The Court emphasised that the applicant expected his activities to 
remain private and that his identity would not be disclosed. While it 
accepted that the applicant did not conceal his dynamic IP address, 
it also underlined that this could not be decisive in assessing whether 
his expectation of privacy was reasonable from an objective point of 
view. On that point, it reiterated that anonymity on the internet is 
an important component of the right to privacy and that a dynamic 
IP address, even if visible to other users of the network, cannot be 
traced back to the specific computer without verification by the ISP, 
before concluding that it was sufficient to note that the secrecy of 
correspondence required that any interference with this right be 
based on a court order: ‘Therefore, also from the standpoint of the 
legislation in force at the relevant time, the applicant’s expectation 
of privacy with respect to his online activity could not be said to be 
unwarranted or unreasonable.” 


The European reasonable expectation of privacy doctrine could have 
various implications for synthetic technologies, three of which stand out: 


First, when a person has made information about themselves public, 
this does not mean that others can process that information for their 
own purposes. This contrasts with the rule in the GDPR, under 
which sensitive data may be processed if those data have been mani- 
festly made public by the data subject, although even in that case, 
the purpose limitation principle would apply. The doctrine under the 
ECHR means that it is not a given that a HR or an AR user may 
harvest data about a person found on the internet to make decisions 
or to project as visual cues. Also, when a person publishes informa- 
tion about themselves, such as a picture on an Instagram account, 
this does not mean it can be used for creating a DF. In all of these 
cases, the person would have a reasonable expectation of privacy and 
the Court would assess each case on its merits, provided that the de 
minimis rule is met. 

Second, even in public and professional settings, people have a 
reasonable expectation of privacy. This means that the argument that 
a person cannot object to their data being recorded by an AR device 
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or HR in the professional or public sphere would not be accepted by 
the ECtHR. 


Third, the ECtHR grants a reasonable expectation of privacy even to 
people downloading and distributing child pornography. This means 
that it will also do so with respect to people creating and distributing 
DF non-consensual pornography and DF pornography of non- 
existent children. 


It should be stressed that in each of these cases, the fact that those 
involved have a legitimate claim to privacy under Article 8 § 1 does not 
mean that their right cannot be curtailed if there are legitimate reasons 
to do so (Article 8 § 2). 


Legitimate Expectation of Privacy 


The legitimate expectation of privacy doctrine was first accepted in 
a case in which Caroline von Hannover, the Princess of Monaco, had 
spent more than 10 years in unsuccessful litigation in German courts. 
She alleged that as soon as she left her house, she was hounded by 
paparazzi who followed her every movement, be it crossing the road, 
collecting her children from school, shopping, going out for a walk, 
engaging in sport activities or going on holiday. The ECtHR consid- 
ered that anyone, even if they are known to the general public, must 
be able to enjoy a ‘legitimate expectation’ of privacy. It emphasised 
the importance of the freedom of expression and the role of the press, 
but also found that in this case, the press had violated the Princess’ 
right to privacy.” 

In a subsequent case, the Court went one step further, as it observed 
that a press publication had been dealing with rumours about a poli- 
tician’s private and family life and stressed that politicians also have 
a ‘legitimate expectation of privacy’. It distinguished a politician’s 
alleged marital problems from their state of health, which, although 
belonging to the personal sphere, could have a bearing on the exercise 
of their duties. Because the couple’s private life had not played a role 
during the politician’s second term in office and because the alleged 
extramarital relationship of the politician’s wife did not have any link 
with their public duties and responsibilities, the Court found that 
the publication did not contribute to any public debate in respect of 
which the press has to fulfil its role of ‘public watchdog’, but merely 
served to satisfy the curiosity of a certain readership. The fact that the 
alleged extramarital affair of the politician’s wife was with the leader 
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of the extreme right or neo-fascist party, which on many accounts, if 
true, could be deemed politically sensitive, did not alter this fact.” 


e The Court condoned a conviction under criminal law of a former 
girlfriend of a Prime Minister who wrote an autobiographical book 
about their relationship. It stressed that the facts set out in the book 
were not in dispute and were presented in a compassionate manner, 
and that the style was not provocative or exaggerated. The Prime 
Minister was clearly a public figure and he had even consented to his 
photograph being used on the cover of the book.” The Court stressed 
that even though the emphasis in the book was on the applicant’s 
private life, it nevertheless contained elements of public interest and 
that most of the information concerning the Prime Minister’s private 
life had already been widely disclosed. Despite all these considera- 
tions, it found the conviction under criminal law to be reasonable, 
because some information in the book concerned his intimate life, 
which had not been disclosed previously. 


e The Court took a next step when a broadcasting company aired a 
television programme named ‘Jungle’, in which three videos that had 
been filmed with a hidden camera were shown. In the first video, AC, 
then a Member of Parliament and chairman of the committee on elec- 
tronic gambling, was shown entering a gambling arcade and playing 
on two machines. The second video showed a meeting between AC 
and associates of the television host of ‘Jungle’, MT, during which 
the first video was shown to AC. The third video showed a meet- 
ing between AC and MT in the latter’s office. Although the ECtHR 
observed that the publication was not without political importance, 
it stressed that the parliamentarian was not informed about the fact 
that they were being filmed and was unconvinced that secret filming 
was necessary for the second and third videos. Consequently, it found 
that the sanction of €100,000 was not only in conformity with the 
Convention, but was also lenient, given the legitimate expectation of 
privacy of AC.” 


e A final example is a case about a video of an applicant, a well-known 
investigative journalist, in which she was shown having sex with her then 
boyfriend, which was leaked to the press. The newspaper in question 
did not publish the material, but merely referenced it and commented 
on it in a negative manner. The government argued that the applicant 
had no legitimate expectation of privacy, not only because she was a 
public figure, but also because the material that was referenced in the 
report was already made public. The Court disagreed: 
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It is true that, once a person’s privacy has been breached and the informa- 
tion about it has entered into public domain, the damage is already done, 
and it is virtually impossible to restore the situation to when the breach 
had never happened. However, while responsible reporting on matters of 
public interest in accordance with the ethics of journalism is protected by 
the Convention, there can be no legitimate public interest in exploiting 
an existing breach of a person’s privacy for the purpose of satisfying the 
prurient curiosity of a certain readership, publicly ridiculing the victim and 
causing them further harm.” 


The ‘legitimate expectation of privacy’ doctrine has several implications 
for synthetic technologies, three of which stand out: 


First, although public figures have a limited expectation of privacy, 
this is far from non-existent. This section has focused on cases in 
which the right to privacy of public figures prevailed over the freedom 
of expression of media. Although there are at least as many cases in 
which the ECtHR gave an opposite decision, it illustrates that there 
are limits to the freedom of expression vis-a-vis public figures. This 
has important repercussions for DFs of politicians. 


Second, the fact that a person has not published or made available 
information about a public figure that violates the legitimate expec- 
tation of privacy themselves, but only re-uses it or refers to it, does 
not mean that they are absolved from their obligations under the 
Convention. This has important repercussions when fake or illegiti- 
mate information is re-used or shared, for example, if an existing DF 
that violates a person’s right to privacy is retweeted and commented 
on, or an AR projects slanderous information next to the face of a 
person the user encounters. 


Third, granting a broader level of protection to civil servants and 
politicians under the current regulatory framework in light of 
synthetic technologies could be considered. The contemporary 
approach is understandable because, on the one hand, politicians 
seek public office themselves and, on the other hand, it is vital for 
a democracy that their actions, policies and behaviour are meticu- 
lously scrutinised. However, many politicians decide to leave office 
and others decide not to run for office because of the toxic online 
environment. Synthetic technologies have not created this problem, 
but may well exacerbate it. It is already difficult for many in office 
to deal with hateful comments on X and other platforms, but for 
a politician to know that not only their children but also all their 
children’s classmates will check out the latest DF pornography in 
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which they feature may simply be too much. Consequently, addi- 
tional protection may be necessary to ensure that the most qualified 
people will continue to seek office. The extent to which this is desir- 
able should be a matter of societal debate, as stronger protection of 
rights for public figures may come at the expense of citizens’ rights 
to satire and parody, and even of the possibility to express critical 
ideas about those in office. 


Freedom of Expression and Misinformation 


Section 5.5.4 will shed more light on the liability of internet providers that 
provide a platform for distributing disinformation and the various EU 
initiatives that target the spread of fake news. With respect to Article 10 
ECHR, aside from the specific points raised by synthetic technologies 
vis-a-vis the doctrines of the reasonable and legitimate expectation of 
privacy, there are two potential regulatory gaps. 

On the one hand, under the current regime, making incorrect, 
false or misleading statements is not prohibited as such. It is prohib- 
ited if a speech act causes harm to a personal or general interest. 
However, with synthetic technologies, there may be no direct harm. 
The harm may only materialise in the long term, or it may be difficult 
to establish the exact causal relationship between the speech act and 
the potential harm caused. In particular, the current focus on harm 
before a matter can be addressed under the human rights framework 
entails three things: 


e A series of complex issue may arise with regard to the causal relation- 
ship between an untrue, incorrect or misleading statement and the 
damage it causes (anticipated or actual). 


e Untrue, incorrect or misleading statements are not regulated or 
prohibited as such, yet they can be problematic. For example, fake 
satellite images may be produced in which Russia appears to be 
moving its nuclear missiles to the Latvian border, causing political 
tension, or fake news could be spread about vaccines, leading to a 
decline in vaccine take-up. A political leader may distribute a video 
that makes it appear that thousands of supporters are at their rallies 
when in fact there are only a handful. 


e Fake information contributes to a post-truth world: a world in which 
facts and opinions, truths, lies and half-truths are increasingly diffi- 
cult to distinguish and disentangle. 
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Regulation could therefore be adopted on this point, for example, 
by prohibiting clearly untrue, incorrect or misleading statements. 
However, this would raise the question of what is ‘true’ and who 
decides on that matter; do we want the government to have a monopoly 
on truth? Still, judges already have to make such decisions under the 
current legal paradigm, such as when it comes to the interpretation 
of the data quality principle or when a person claims to be defamed 
by slanderous expressions. In addition, many European countries have 
adopted rules on Holocaust denial. Thus, although difficult questions 
may emerge, setting rules and standards for untruthful expressions is 
not unprecedented. 

On the other hand, regulating misinformation in light of democratic 
elections may be necessary. Several American states have already adopted 
specific provisions on the use of DFs in election campaigns, inter alia, in 
the light of Russian attempts to influence election results.” It is impor- 
tant to consider that it is not only foreign powers that may spread fake 
news during election campaigns: national and domestic political parties 
may do so as well. Several options could be considered. 


e Special tort action could enable politicians or political parties who 
have fallen victim to DFs during election campaigns to claim damages. 


e Ex parte proceedings for taking down misleading content could be 
considered. 


e Criminal action could be considered with respect to spreading false 
and misleading information during election campaigns with the 
purpose of influencing election results. 


However, this may be a sensitive topic because what is considered to be 
true or false is often the nexus of the political debate. 

Importantly, criminal and civil law provisions only have a limited 
practical effect on the threat of foreign election interference. Other 
instruments should be considered there, such as diplomacy or possible 
digital counter-measures. Legislative changes should not only be consid- 
ered with respect to countries influencing Western democracies; countries 
like Russia, China and Iran are also targeting the Global South. This is 
being done for a wide variety of reasons, such as influencing concrete 
decision making (eg, so that a Russian state-owned company gets a 
contract instead of an EU-based company), influencing local elections 
(eg, to bring a China-friendly regime to power) or influencing decisions 
at the international level (eg, by getting a country to vote in favour of the 
lifting of sanctions against Iran). 
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The EU has announced a Defence of Democracy Packages: 


The package will include a legislative initiative strengthening the resilience 
of the EU democratic space to foreign interference (to curb the influence 
exerted through covert interest representation services paid for or directed 
from outside the EU) and other non-legislative support measures. These are 
expected to include a non-legislative initiative to complement the legislative 


initiative on foreign interference, a recommendation on secure and resilient 


elections, and a recommendation on civic engagement.”® 


It is currently unclear how that package will take shape and whether the 
measures contained in it will be sufficient to tackle the various threats 
for democracy and the rule of law synthetic technologies entail.””? The 
Council of Europe has not adopted a comprehensive approach to coun- 
tering online disinformation and the impact on democratic elections,*° 
although it is aware of the many dangers.*! 


5.3.6. Conclusion 


This section explored how freedom and property rights accepted under 
the ECHR and the CFREU may apply to synthetic technologies. In 
particular, it focused on open-ended questions and the regulatory gaps 
that may emerge: 


e The current human rights catalogue seems ill-suited to be applied 
to non-human entities; if, in the future, rights were to be attrib- 
uted to non-human entities, a new catalogue should be drawn up, 
which distinguishes between deceased persons, avatars, non-existent 
persons and robots, assigns new rights and freedoms to each of these 
entities, decides on their legal standing and on the question of legal 
personhood, and takes into account their current and future (desired) 
moral status. 


e The right to physical and intellectual property, as well as the 
freedom to conduct a business, could give rise to a number of 
complicated conflicts — for example, if there are two contrasting 
property rights, if the authorship of a synthetic work is difficult 
to assign because there are multiple parties involved, if a HR is 
destroyed and its owner claims loss of future possession or if one 
party has intellectual property rights over specific content, but the 
other invokes the fair use doctrine, the private copy exemption or 
the right to parody. 
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With respect to the right to privacy, the ECtHR has reinterpreted this 
doctrine as a personality right. This means that almost any inter- 
ference with an interest of a natural person will be considered to 
interfere with Article 8 ECHR. This may include harm to non-human 
entities owned or operated by a natural person. However, the extent 
to which this will be accepted, especially in light of the de minimis 
requirement, is unclear. In addition, when harm to a deceased person 
affects the interests of a natural person, the latter may invoke their 
right to privacy. There is ongoing debate, which is particularly impor- 
tant in the light of synthetic technologies, regarding whether the 
interests of dead people should also be protected through a so-called 
‘post-mortem privacy right’. 

The doctrine of the chilling effect may set limits on the use of synthetic 
technologies when people adjust their behaviour when they know 
or have reason to believe that their data will be gathered, for exam- 
ple, by a person wearing an AR device or a HR they may encounter. 
However, the issue of whether a chilling effect exists is a particularly 
difficult one to address because it depends on context and personal 
evaluations. This will need to be determined on a case-by-case basis. 
It is clear that under both the Council of Europe’s and the EU’s legal 
acquis, there is room for stronger rules on preventing or mitigating 
the risks of manipulative AI and safeguarding individual autonomy 
and psychological integrity. 

On many accounts, the current anti-discrimination doctrine is ill- 
suited for tackling the danger of biases fed to and generated by AI 
systems. When Al is trained on biased data sets, it will have an incom- 
plete, incorrect or distorted view of reality. When AI predictions are 
inaccurate, they nevertheless shape and change existing reality. A 
revised framework that is tailored to AI may need to focus on how 
data are gathered and analysed, and on the procedure running up to 
a decision rather than on the decision itself. 


There are cases in which the freedom of expression and the right to 
privacy will clash. The reasonable expectation of privacy entails that 
people can legitimately invoke the right to privacy both in private 
and in public and professional settings, as well as when they have 
made information public themselves. The legitimate expectation 
of privacy means that although public figures must tolerate more 
substantial intrusions into their private lives, their right to privacy 
cannot be completely disregarded. However, where the boundary 
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is drawn with respect to the various uses of synthetic technologies 
is unclear. Three regulatory questions have been discussed: first, 
whether untruthful and incorrect expressions should be prohib- 
ited, even if they do not cause direct harm to personal or societal 
interests; second, whether additional regulation is needed against 
influencing democratic elections and political decision making 
through the use of fake news; and, third, whether public figures and 
politicians should be granted a higher level of protection to ensure 
that those best qualified for public office actually seek that office. 
Each of these potential legislative changes triggers a number of 
complicated moral and legal questions. 


5.4. DATA AND TECHNOLOGY REGULATION 
5.4.1. Introduction 


This section explores EU regulation in the context of AI, technol- 
ogy and interoperability. First, it discusses the various enacted and 
proposed legal instruments setting technical requirements for develop- 
ing, marketing and exporting digital products (section 5.4.2). This is 
followed by a discussion of the proposed AI Act (section 5.4.3). Then, 
the various rules on data control and data portability are evaluated in 
light of synthetic technologies (section 5.4.4), and the tenability of the 
underlying philosophy of Europe’s data regulations in an Al-mediated 
world is discussed (section 5.4.5). Finally, a short conclusion is provided 
(section 5.4.6). 


5.4.2. Product Requirements 


The EU has adopted and proposed various rules for manufacturing, 
utilising and exporting products (digital and physical) and machines,* 
such as, but not limited to: 


e The proposed Machine Regulation® subjects high-risk machines to a 
conformity check. In determining the probability and severity of harm, 
the Regulation stresses, account should be taken of the degree to which 
each affected person would be impacted by the harm, the number of 
persons potentially affected, the degree to which potentially affected 
parties are dependent on the outcome produced by the machinery 
product, the degree to which they are in a vulnerable position vis-a-vis 
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the user of the machinery product, the degree of reversibility of the 
harm produced by the machinery product, the degree to which the 
machinery product has been used for a specific purposes, and indica- 
tions of harm that have been caused in the past by machinery products 
that have been used for a specific purpose. The rules contained in this 
Regulation will be relevant in particular for HRs used in factories. 


The Regulation on Medical Devices* lays down additional require- 
ments for the production of medical devices. These may be relevant 
when HRs, AR devices and VR worlds are deployed in medical facili- 
ties, for treatment purposes or as care robots in homes. A medical 
device is defined broadly as any instrument intended by the manu- 
facturer to be used for human beings for the purpose of diagnosis, 
prevention, monitoring, prediction, prognosis, treatment or allevia- 
tion of disease or disability. The Regulation provides that they must 
be designed and manufactured in such a way that, when used under 
the conditions and for the purposes intended, they will not compro- 
mise patient safety. Manufacturers must eliminate or reduce risks as 
far as possible (inherently safe design and construction) and, where 
appropriate, take adequate protection measures. If the devices are 
intended to administer medicinal products, they must be compatible 
with the medical products, and their performance should be main- 
tained in accordance with the intended use. Devices with a measuring 
function must be designed and manufactured in such a way as to 
provide sufficient accuracy. 


The proposed Regulation on Product Safety% underlines that only 
safe products can be put on the EU market. These are products that, 
under normal or reasonably foreseeable conditions of use or misuse, 
do not present any risk or only minimal acceptable risks, and are 
consistent with the high level of protection of health and safety of 
consumers. Risk is defined under the Regulation as the combination 
of the probability of an occurrence of a hazard causing harm and the 
degree of severity of that harm. Comparatively, high risk is defined 
as a risk for which the combination of the probability of occurrence 
of a hazard causing harm and the degree of severity of the harm 
is considered to require rapid intervention by market surveillance 
authorities. Importantly, a recital to the Regulation as proposed by 
the Commission also referred to ‘psychological risk, development 
risks, in particular for children, mental risks, depression, loss of sleep, 
or altered brain function’. This interpretation of safety might have 
had serious repercussions for, inter alia, VR games that are addictive 
or lead to reduced social contact, sleep deprivation and depression. 


142 Law 


However, this recital was struck from the current wording of the text, 
so it remains to be seen whether such effects will be considered to fall 
under the scope of this Regulation, which mainly focuses on ensur- 
ing physical safety. 

The Directive on the Safety of Toys has a limited scope.** It covers, 
inter alia, playground equipment intended for public use and auto- 
matic playing machines. The Directive might apply to AR and VR 
devices used in arcade-like settings, which would mean manufac- 
turers and providers would have to ensure that the toys do not 
jeopardise the safety or health of users or third parties when they 
are used as intended or in a foreseeable way, bearing the behaviour 
of children in mind. Harm can both refer to physical injury and 
any other damage to health, including long-term health effects. 
Although the Directive’s focus is on physical dangers (eg, radioac- 
tivity, inflammation, strangulation, electrocution and suffocation), 
it cannot be excluded that it might also apply if there are adverse 
mental health effects. 


The Network and Information Systems (NIS2) Directive encour- 
ages states to draw up elaborate laws and policies in the domain of 
cybersecurity.*” 

The Cyber Security Act focuses on networks and information systems, 
and sets up a regulatory body to take the regulatory initiative in this 
field. The European Union Agency for Cybersecurity (ENISA) has 
drawn up a report on cybersecurity issues in relation to AI, in which 
it maps various Al-related security risks through the Confidentiality, 
Integrity and Availability (CIA) model. 


Table 5.4 ENISA’s application of the CIA paradigm in the context of AI*? 


Security goal Al-specific attacks) 


Contextualisation in AI (selected examples of 


Confidentiality Oracle: A type of attack in which the attacker explores 


a model by providing a series of carefully crafted inputs 
and observing outputs. These attacks can be precursor 
steps to more harmful types, for example, evasion or 
poisoning. It is as if the attacker made the model talk in 
order to then better compromise it or to obtain informa- 
tion about it (eg, model extraction) or its training data 
(eg, membership inference attacks and inversion attacks). 


(continued) 
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Table 5.4 (Continued) 


Security goal Al-specific attacks) 


Contextualisation in AI (selected examples of 


Model disclosure: This threat refers to a leak of 

the internals (ie, parameter values) of the Machine 
Learning model. This model leakage could occur 
because of human error or a third party with too low 
a security level. 


Integrity Evasion: A type of attack in which the attacker works 


on the ML algorithm’s inputs to find small perturba- 
tions leading to a large modification of its outputs (eg, 
decision errors). It is as if the attacker created an opti- 
cal illusion for the algorithm. Such modified inputs are 
often called adversarial examples. 


Poisoning: A type of attack in which the attacker 
alters data or models to modify the ML algorithm’s 
behaviour in a chosen direction (eg, to sabotage its 
results or to insert a back door). It is as if the attacker 
conditioned the algorithm according to its motivation. 


Availability Denial of service: ML algorithms usually consider 


input data in a defined format to make their predic- 
tions. Thus, a denial of service could be caused by 
input data whose format is inappropriate. However, 

it may also occur that a malicious user of the model 
constructs an input data (a sponge example) specifi- 
cally designed to increase the computation time of the 


model and thus potentially cause a denial of service. 


There is a Cyber Resilience Act” under discussion, which, if adopted, 
would regulate products with digital elements, namely software or hard- 
ware products and their remote data processing solutions. Remote data 
processing includes any data processing performed at a distance for 
which the software is designed and developed by the manufacturer or 
under the responsibility of the manufacturer, and the absence of which 
would prevent the product from performing one of its functions. The Act 
distinguishes between ordinary products with digital elements, critical 
products with digital elements and highly critical products with digital 
elements. When determining the level of cybersecurity risk, account 
must be taken of the product’s cybersecurity-related functionality, its 
intended use in sensitive environments, its intended use in performing 
critical or sensitive functions, the potential extent of an adverse impact 
and the extent to which the use of products has already caused material 
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or non-material loss. All products must be designed, developed and 
produced in such a way that they ensure an appropriate level of cyberse- 
curity based on the risks. If there are known exploitable vulnerabilities, 
the products cannot be marketed. It must be ensured that the products 
are secured, the data are encrypted, the processing of data remains 
limited to what is necessary and the integrity of the device is safeguarded. 
These rules are important, inter alia, in relation to synthetic technologies 
produced in totalitarian countries that have back doors built into them.”! 


e This theme is also addressed by the Dual Use Regulation, but in 
reverse.” Dual-use items are technologies that can be used for good 
(eg, monitoring and arresting terrorism suspects) but also for bad 
(eg, monitoring and subsequently quelling political dissidents). The 
Regulation contains export control rules, including a set of assessment 
criteria and types of authorisations, a list of dual-use items, provisions 
for end-use controls on non-listed items, and specific control measures 
and compliance to be introduced by exporters, such as record-keeping 
and registers.” The Regulation only covers the export of technologies 
from EU to non-EU territories. At a minimum, such limitations should 
prevent dictatorial regimes from obtaining technologies that they can 
misuse. Importantly, the Regulation also places emphasis on the poten- 
tial human rights effects of exported technologies. With regard to 
cybersurveillance items, the Regulation requires competent authorities 
to consider in particular the risk of them being used in connection with 
internal repression or the commission of serious violations of human 
rights and international humanitarian law. The Regulation includes 
broad references to export controls on ‘Electronics’, ‘Computers’ and 
“Telecommunications and information security computers’. Each of 
those categories is further specified in the Regulation in a very detailed 
list of sensors, operational capacities, and hardware and software. The 
extent to which synthetic technologies are covered by those lists needs 
to be determined on a case-by-case basis and depends, inter alia, on the 
software, microchips and sensors being used. European countries may 
be under a positive obligation to prohibit the export of technologies if 
they know technologies are likely to be misused by regimes that have a 
bad track record with respect to human rights. 


e Finally, a European Commission expert group has designed the Ethics 
Guidelines for AI, which includes a Framework for Trustworthy AI.’ 
This framework suggests that AI must be lawful, ethical and robust. 
Robustness entails that AI systems should perform in a safe, secure and 
reliable manner, and that safeguards should be foreseen to prevent any 
unintended adverse impacts. It stresses that of utmost importance is the 
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respect for human dignity, which entails that all people are treated with 
respect due to them as moral subjects rather than merely as objects to 
be sifted, sorted, scored, herded, conditioned or manipulated (see also 
Chapter 4, section 4.3). This is why, the Guidelines suggest, AI systems 
should be developed in a manner that respects, serves and protects 
humans’ physical and mental integrity, as well as their personal and 
cultural sense of identity and the satisfaction of their essential needs. 
The Guidelines stress the importance of the rule of law, respect for demo- 
cratic processes, human freedom and autonomy, and the prohibition of 
discrimination. In terms of ethics, it lays down four leading principles: 


o Human autonomy: humans that interact with AI systems must be 
able to keep full and effective self-determination over themselves. 
Consequently, AI systems should not unjustifiably subordinate, 
coerce, deceive, manipulate, condition or herd humans. On the 
contrary, they should be designed to augment, complement and 
empower human cognitive, social and cultural skills. 

o Prevention of harm: Al systems should protect human dignity and 
mental and physical integrity. Consequently, AI systems should not 
be open to malicious use, and vulnerable persons should receive 
greater attention and be included in the development, deployment 
and use of AI systems. Preventing harm also entails preventing 
information asymmetries. 

o Fairness: fairness, from a substantive point of view, entails the obli- 
gation to ensure the equal and just distribution of both benefits and 
costs, and ensuring that individuals and groups are free from unfair 
bias, discrimination and stigmatisation, as well as from being deceived 
or unjustifiably impaired in their freedom of choice. Procedurally, 
fairness entails the ability to contest and seek effective redress against 
decisions made by AI systems and by the humans operating them. 

o Explicability: processes should be transparent. The capabilities 
and purpose of AI systems should be openly communicated, and 
decisions should be explainable to the parties that are affected 
both directly and indirectly. 


In terms of technical robustness, it further distinguishes between 

several sub-principles: 

o Resilience to attack and security: Al systems should be protected 
against vulnerabilities that can allow them to be exploited by 
adversaries (eg, hacking). 

o Fallback plan and general safety: Al systems should have safe- 
guards that enable a fallback plan in case of problems (eg, allowing 
a switch from a statistical to a rule-based procedure). 
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o Accuracy: AI systems must be able to make correct judgements 
to correctly classify information into the proper categories, for 
example, and to make correct predictions, recommendations or 
decisions based on data or models. The guidelines emphasise that 
a high level of accuracy is especially crucial in situations where the 
Al system directly affects human lives. 

o Reliability and reproducibility: AI systems should be reproducible. 
This enables scientists and policy-makers to accurately describe 
what AI systems do. Replication files can also facilitate the process 
of testing and reproducing behaviours. 


Framework for Trustworthy AI 


Trustworthy AI 
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Operationalise the key requirements Tailor this to the specific AI application 


Figure 5.3 Framework for trustworthy AI’ 


What is clear from the foregoing is that synthetic technologies must be 
developed and designed in a way that they, inter alia, only process data 
when necessary, that these data must be adequately protected against 
misuse and third-party hacks, that the technologies may not do physical 
harm, that they must have safeguards in place against bias and discrimina- 
tory effects, that they produce results that are accurate and reproducible, 
and that there are systems in place for when things go awry. However, 
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there are many open-ended questions because many of these regimes have 
only recently been adopted or proposed, their primary goal is not regu- 
lating synthetic technologies and many of the principles are general in 
nature. These points include, but are not limited to, the following: 


e Many of these instruments are sectoral or concern niche use of prod- 
ucts, such as in arcade settings. Although sectoral regulation can be 
useful, three things stand out: 


o First, an overall framework for product requirements is mostly 
lacking. 

o Second, although some sectors and use cases are specifically 
addressed, other important sectors and areas are not. 

o Third, sectoral regulation can be valuable for specifying in detail 
what the general principles from an overarching framework entail 
for a specific setting; however, the regulations on product require- 
ments as discussed remain mostly abstract and set out general duties 
of care. There can be substantial differences in how the transpar- 
ency principle or how the requirement of technical security should 
be understood and implemented according to sector or context; yet, 
the current regimes do not answer questions such as: what does it 
mean to be transparent? Which information needs to be communi- 
cated to whom and how? What does it mean to implement technical 
security standards? Which standards must be embedded in which 
systems and how? What does it mean to have a fair system? Which 
definition of fairness is used and how should it be implemented? 


e It is unclear what these general rules would mean precisely for the 
various synthetic technologies, as they are written either for AI in 
general or, even more broadly, for products and machines. 


e Several instruments primarily focus on physical harm, while most 
synthetic technologies will result in immaterial personal harm or 
societal harm, if any. It is unclear to what extent such harm will also 
be covered by these instruments. 


¢ The rules on the export of technology are general and vague, and are 
not tailored to synthetic technologies. Given the potential impact of 
these technologies on human rights, the rule of law and democracy, 
it can be questioned whether these technologies should be regulated 
explicitly in the Dual Use Regulation. 


e Rules on the import of technologies are mostly absent. Since most AI 
technologies are developed outside Europe and imported, developing 
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an equivalent of the Dual Use Regulation for importing technologies 
that are important into Europe could be considered. 


Finally, although the framework for trustworthy AI is elaborate, it is 
not legally binding. 


5.4.3. AI Regulation 


An AI Act” is currently under discussion in the EU. If this Regulation 
were to come into effect in its current form, there would be three levels 
of AI systems: (1) low-risk AI systems; (2) high-risk AI systems; and 
(3) prohibited AI systems. 


For the first category, no additional rules would be imposed. 


With respect to the third category, the Act states that the following are 
prohibited” — the placing on the market, putting into service or use of 
an AI system: 


that deploys subliminal techniques beyond a person’s consciousness 
in order to materially distort a person’s behaviour in a manner that 
causes or is likely to cause that person or another person physical or 
psychological harm; 


that exploits any of the vulnerabilities of a specific group of persons 
due to their age, physical or mental disability, in order to materi- 
ally distort the behaviour of a person pertaining to that group in a 
manner that causes or is likely to cause that person or another person 
physical or psychological harm; 

by public authorities for the evaluation or classification of the trust- 
worthiness of natural persons over a certain period of time based on 
their social behaviour or known or predicted personal or personal- 
ity characteristics, with the social score leading to either detrimental 
or unfavourable treatment of natural persons or groups (a) in social 
contexts which are unrelated to the contexts in which the data was 
originally generated or collected, or (b) that is unjustified or dispro- 
portionate to their social behaviour or its gravity; 


that take the shape of ‘real-time’ remote biometric identification 
systems in publicly accessible spaces for the purpose of law enforce- 
ment, unless and insofar as such use is strictly necessary for law 
enforcement purposes. 
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Since the Act has not been finalised yet, it is unclear how these rules 
would map onto synthetic technologies, inter alia, because there are still 
open questions regarding what would amount to ‘physical or psychologi- 
cal harm’, ‘materially distorting a person’s behaviour’ or ‘unfavourable 
treatment’. Still, itis clear that, in particular, two applications of synthetic 
technologies may be prohibited under the Act: 


The case in which AR systems are used in smart cities where people 
are subconsciously manipulated. Although it is true that these 
manipulations are so far ‘for the better’ — for example, where AR 
systems are deployed as a means of crowd control to prevent traffic 
jams or to make people in nightlife areas less aggressive — it cannot 
be excluded that these techniques could and would be used for more 
controversial purposes. In addition, it will become clear, presumably 
through jurisprudence on the matter, the extent to which subcon- 
scious manipulation itself amounts to harm, even when it is used ‘for 
the better’. 


The case in which HRs are deployed for law enforcement purposes or in 
which police officers wear AR devices that enable remote biometric iden- 
tification of natural persons. The extent to which this is strictly necessary 
in light of the public interest and is in conformity with the requirements 
of proportionality and subsidiarity would need to be assessed. 


With respect to the second category of AI systems, the Act sets out a 
large number of categories of high-risk AI systems,!° such as: 


AI systems intended to be used for the ‘real-time’ and ‘post’ remote 
biometric identification of natural persons; 


Al systems intended to be used for the purpose of determining access 
or assigning natural persons to educational and vocational training 
institutions, for assessing students or for assessing participants in 
tests required for admission to educational institutions; 


AI systems intended to be used for recruitment or selection of natu- 
ral persons or for making decisions on the promotion, assignment, 
distribution and termination of work; 


AI systems intended to be used to evaluate the eligibility of natu- 
ral persons for public assistance benefits and services, to evaluate the 
creditworthiness of natural persons or establish their credit score, or 
to dispatch or establish priority in the dispatching of emergency first- 
response services; 
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AI systems intended for making individual risk assessments of natu- 
ral persons, as polygraphs or to detect the emotional state of a natural 
person, to detect DFs or for evaluation of the reliability of evidence, 
for predicting the occurrence or reoccurrence of criminal offences, 
for profiling or for crime analytics; 


AI systems intended to be used as polygraphs or to detect the 
emotional state of a natural person, to assess a risk, including a 
security risk, a risk of irregular immigration, or a health risk, for the 
verification of the authenticity of travel documents or for the exami- 
nation of asylum, visa and residence permit applications; 


AI systems intended to assist a judicial authority in researching and 
interpreting facts and the law, and in applying the law to a concrete 
set of facts. 


It is obvious that many synthetic applications discussed in Chapter 2 may 
fall into these categories, as they are used for a wide variety of ways, 
including in educational settings, for recruiting, for risk assessments 
and for security purposes. In addition, the use of AI for DF detection is 
explicitly mentioned. 


For all the systems falling into the second category, the AI Regulation 


lays down obligations and requirements that need to be respected. Some 
of these include the following: 


A risk management system needs to be established, consisting of a 
continuous iterative process run throughout the entire lifecycle of 
an AI system, requiring regular systematic updating and comprising 
an identification and analysis of known and foreseeable risks. 1! 


When training models, data should be kept on the design choices, 
data collection and relevant data preparation, which should include 
an examination in view of possible biases and the identification of 
any possible data gaps or shortcomings. 


A system should enable logging, for example, to record each use of 
the system, the reference database against which input data has been 
checked by the system and the input data for which the search has led 
to a match. 


A system should be designed and developed so that natural persons 
can effectively oversee it while the AI system is in use. 


The manufacturer and the provider of the product are each responsible 
for compliance with these rules, which, as the European Commission 
has explained, should be understood inter alia as a safeguard against 
self-learning AI systems that go rogue. 
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e Before placing a high-risk AI system on the market: 


© importers of such systems have to ensure that the appropriate 
conformity assessment procedure has been carried out by the 
provider of that AI system, that the provider has drawn up the 
required technical documentation and that the system bears 
the required conformity marking; 

o distributors have to verify that the system bears the required 
conformity markings, that it is accompanied by the required 
documentation and instruction of use, and that the provider 
and the importer of the system have complied with their 
obligations. 


e Any distributor, importer, user or other third party shall be consid- 
ered a provider. This means they have to ensure that their high-risk AI 
systems are compliant with the various requirements, have a quality 
management system in place, draw up the technical documentation, 
keep logs, do a conformity assessment, take the necessary correc- 
tive actions when a breach is established and inform the supervisory 
authorities thereof. 


e Users of high-risk AI systems should use systems in accordance with 
the instructions for their system and monitor the operation of the 
high-risk AI system on the basis of the instructions. Users also keep 
the logs that are generated by that high-risk AI system. 


Finally, certain AI systems are subject to a transparency obligation.!” If 
Al systems are intended to interact with natural persons, those persons 
should be informed that they are interacting with an AI system, unless 
this is obvious from the circumstances or when law enforcement authori- 
ties use AI systems to detect, prevent, investigate and prosecute criminal 
offences. This means that should HRs become truly indistinguishable 
from humans, they should let natural persons with whom they interact 
know that they are non-human, for example, when used for hospitality 
purposes. Users of emotion recognition systems or biometric categori- 
sation systems are obligated to inform those affected of the operation 
of the system, except when law enforcement authorities operate these 
systems. This would mean that when AR devices are equipped with 
emotion recognition systems, this should be communicated to those 
that are encountered. Users of AI systems that generate DFs have the 
obligation to disclose that the content has been artificially generated 
or manipulated, except in the situation in which law enforcement 
authorities do so or when this would conflict with citizens’ freedom of 
expression. 
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How the transparency obligation for DF technology would work in 


practice is not entirely clear: 


The provision seems to cover all manipulated content; however, the 
problem is that, depending on the definition of manipulation, any 
communication technology distorts reality: video services have built- 
in tools that smoothen skin tones, audio services automatically filter 
out treble and so forth. Some experts estimate that, in five years, more 
than 90 per cent of all online content will be manipulated in some 
form. Importantly, the AI Act also refers to manipulated content 
about objects, places and events. Would this provision consequently 
also apply to a smiley-faced sun? 

The provision only requires users to disclose that the content has 
been manipulated, but to whom should they disclose this informa- 
tion: the general public, the person depicted or the platform on which 
it was posted? 

It is the user of the Al system who is obligated to report informa- 
tion about the manipulation, but how should they do so: through 
the metadata attached to the DF, in the DF itself or in the descrip- 
tion of the DF? The AI Act only states ‘shall disclose’, while 
the recital speaks of ‘labelling’. Labelling is often used within 
the AI context for preparing data by labelling: categorising the 
data. This would involve metadata being added, which would 
most likely not inform the general public that content has been 
manipulated, unless they have an automatic manipulation detec- 
tion program running on their computer, which would in itself 
qualify as a high-risk system. However, this would allow plat- 
forms like Facebook and Instagram to easily block manipulated 
content. The emphasis on metadata and labelling has the poten- 
tial drawback that real content that has been made to appear 


manipulated by adding traces of manipulation in the metadata 
will also be blocked. 

What should that information include: should it merely state ‘this 
content has been manipulated’ or should it provide a description of 
exactly what has been manipulated and how? 

How does this provision relate to the GDPR and in particular the 
transparency obligation contained therein? 


In addition, the following questions emerge with respect to the applica- 
bility of the AI Act in general to synthetic technologies: 
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The Act contains a rather limited list of prohibited AI systems; the 
question is whether the list should be longer, given the danger of AI 
systems in general and the risks to human rights, the rule of law and 
democracy several synthetic technologies pose. 


The AI Act mainly adopts a risk-based approach. This entails two 
potential pitfalls, which may play out simultaneously: 


o It may stifle innovation. Because of the administrative and bureau- 
cratic costs, it will most likely be larger corporations rather than 
innovative start-ups that can uphold these rules and because these 
bigger parties prefer safety. Risk-based regulation is open and vague; 
parties do not get clear guidance from the regulator on what is and 
what is not allowed. Strict rules often boost rather than stifle inno- 
vation, because companies know within which boundaries they can 
operate and can thus make stable long-term investments. 

o The evaluation of potential risks is initially left up to private organisa- 
tions, only later to be assessed by a regulatory or court. It is known 
that this generally means that the ‘good guys’ will err on the safe side, 
while it will be the ‘bad guys’ who look for loopholes in the meas- 
ures to exploit in a business-friendly way. More generally, the goal 
of impact assessments, under the AI Act, the GDPR and other EU 
data instruments, is that risks will be identified and that if these risks 
cannot be adequately mitigated, the project is cancelled. In prac- 
tice, however, such decisions are seldom made; certain applications 
that are in essence problematic (from a substantive perspective) are 
commonly legitimised by adopting additional procedural safeguards. 


The question is consequently whether the risk-based approach is suit- 
able for addressing the problems at hand. 


A significant problem with data technologies in general and AI in 
particular is the public-private relationship: governments are grow- 
ing ever more reliant on the private sector to operate. This means 
that private parties are directly involved in delivering goods, prod- 
ucts and services in the public interest or are partially involved in 
the execution of public tasks, for example, through public-private 
partnerships. This means that commercial interests will affect the 
public interest, public tasks and the public domain (see Chapter 4, 
section 4.4). This Act does little to reverse this trend or to lay down 
specific safeguards for public-private partnerships, for example, to 
prevent private parties’ access to public data or prevent commercial 
interests from having an impact on the execution of public tasks. 
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e The AI Act mainly sets out procedural and bureaucratic rules in terms 
of risk assessments, keeping records and accountability, and very few 
substantive requirements. This approach mirrors to a certain extent 
that of the GDPR and has two potential advantages: 


o It incentivises an accountability industry, with AI or Data 
Protection Officers, and accountancy firms that specialise in 
compliance checks and procedural guidelines. With respect to 
data protection, this procedural move, combined with the enor- 
mous public attention and the high fines that can be imposed, 
has meant that the matter of compliance has moved from the 
lower echelons of the organisation, such as the IT person, to 
C-level. The AI Act may do the same for the AI safety rules 
contained in it. 

o It is far more easy for supervisory authorities to maintain over- 
sight on and find violations of procedural requirements than with 
respect to substantive rules, as the latter typically involve complex 
legal decisions, while not having done an impact assessment, not 
having appointed a specialist or not having kept records necessary 
is easy to establish. 


Still, one approach does not exclude the other. This would have been 
the opportunity to lay down substantive legal requirements for AI 
systems. One example, among dozens of other substantive require- 
ments that have been suggested in the literature, would be setting 
a percentage of false positives and false negatives that is permis- 
sible according to sector or context. For example, with respect 
to AI systems applied for diagnostic purposes, false negatives 
should be well below 1 per cent and potentially below 0.1 per cent, 
while the ratio of false positives may be higher, but still low 
given the stress caused by a false diagnosis. For law enforcement 
purposes, especially in the field of organised crime and terrorism, 
the permissible level of false negatives also needs to be low, while 
the number of false positives can be higher, given that such systems 
usually indicate which suspects should be on the radar of authori- 
ties rather than leading to legal consequences. For fraud detection 
purposes, to give another example, the percentage of false nega- 
tives may be relatively high, while the percentage of false positives 
should be relatively low, given the impact of false accusations on 
persons’ lives, especially when the tax authorities take automated 
decisions based on AI profiles. 
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5.4.4. Data Control and Portability 


Many European data regulations emphasise the need for data control, 
confidentiality and limits on data transfers, such as, but not limited to, 
the following: 


The GDPR essentially holds that data can only be processed by a data 
controller for predetermined purposes and that the data cannot subse- 
quently be processed by another party for other purposes. No more data 
than necessary for that pre-established goal may be gathered and the data 
that are gathered can be stored no longer than needed for that purposes 
and must be deleted once that goal has been attained. Data must be 
stored confidentially and securely, meaning that they are protected 
against unauthorised access by personnel and third parties (eg, hack- 
ers). In addition, because the GDPR sets the highest level of protection 
in the world, transferring personal data from the EU to, for example, 
China, India or the US, would in principle mean that the data are no 
longer adequately protected. This is why the GDPR provides that doing 
so is prohibited, unless the country to which the data are transferred 
commits to a legal regime essentially equivalent to the GDPR or the 
foreign organisation to which the data are transferred commits to abid- 
ing by the GDPR standards, such as through contractual agreements. !% 


The Data Governance Act! sets rules not only on the processing of 


personal data, but also on the processing of non-personal data.!° 
Non-personal data could entail data that an AR provider collects 
on the environment in which users operate, without linking those 
data to a specific user. For example, if there are sufficient users in the 
area, it could collect real-time data on traffic, weather conditions, 
smog levels and so forth. These data can be valuable and protected 
through, inter alia, intellectual property rights of the synthetic tech- 
nologies providers’ or the operators’ freedom to conduct businesses 
(see section 5.3.2). To ensure their interests are adequately safe- 
guarded, the Data Governance Act protects non-personal data from 
unauthorised access.!° It also stresses that appropriate safeguards 
should be adopted by the non-EU country to which non-personal 
data are transferred. These safeguards should consist of measures 
that ensure non-personal data benefit from a level of protection like 
that applicable in the EU with regard to the protection of trade secrets 
and the protection of intellectual property rights. In addition, re-use 
mechanisms in third countries may need to attach stricter conditions 
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for highly sensitive non-personal data, such as anonymised data sets 
held by actors in the public health system. These rules would have 
considerable repercussions for HRs or AR devices that are used in the 
healthcare sector. 


e The proposed e-Privacy Regulation,” which will replace the 2002 
e-Privacy Directive in time, addresses the processing of electronic 
communications data. These can consist of electronic communica- 
tions content and electronic communications metadata. There are 
three doctrines of relevance for synthetic technologies: 


o Although there are exceptions, the confidentiality principle entails 
that any interference with electronic communications data by 
persons other than the end-users shall be prohibited, such as by 
listening, tapping, storing, monitoring, scanning, surveillance or 
processing of electronic communications data.!” 

o Second, although there are exceptions,!” the use of processing and 
storage capabilities of terminal equipment and the collection of 
information from end-users’ terminal equipment, including about 
its software and hardware, other than by the end-user concerned 
is prohibited, as is the collection of information transferred by 
terminal equipment to enable it to connect to another device or 
piece of network equipment. Like the integrity of one’s home, 
which private parties can only enter with consent or when this is 
necessary (eg, in an emergency), the e-Privacy Regulation recog- 
nises the principle of device integrity. Given that this equipment 
contains information that may reveal details of an individual’s 
emotional, political and social state, third parties are not allowed 
to enter or extract information from the terminal equipment of 
users.!!! Consequently, if AR or VR applications are run on user 
equipment (partially or fully), or if a HR is teleoperated via user 
equipment and, for example, a cookie is used, these rules apply. 

o Third, spam is prohibited.'' How this rule would apply to AR 
and VR settings in which users are confronted with unsolicited 
advertisements of shops in their vicinity or of online virtual shops 
is unclear, but in principle, it is prohibited to communicate directly 
to the consumer using the contact details obtained through the 
provider, unless the user has indicated they want to receive adver- 
tisements of the shops they pass by on their screen. 


e Alongside the EU rules on the transfer of personal and non-personal 
data, the ECtHR has laid down rules for transferring data by 
European governmental (intelligence) agencies from and to foreign 
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counterparts. First, the Court has held that there is the risk that data 
about Europeans are communicated to foreign parties and are treated 
according to a different and more flexible regime than they would 
have been had they stayed in European hands. Second, when data are 
shared with foreign counterparts (for example, data about nation- 
als of that country), they might be used for human rights violations, 
such as imposing the death penalty on or the torturing of political 
dissidents. Third, shared data may be used by European intelligence 
agencies to circumvent the European human rights regime, for exam- 
ple, asking American counterparts to collect data they are not allowed 
to gather themselves.'¥ On all these points, the ECtHR has laid down 
a detailed list of rules, conditions, and checks and balances.!"4 


Consequently, various rules and legal instruments in Europe aim at what 
is sometimes called data privatisation and data nationalisation.!'> These 
principles contrast with the practice on the ground on several points, 
such as, but not limited to, the following: 


While the data regulations require a pre-established purpose for gath- 
ering data, the practice of many organisations is to gather data first 
and only later determine what purpose they could suit. This is not 
only the established practice, one of the promises of algorithms and 
Al is that they can find patterns in data that humans cannot. Thus, so 
the argument goes, it is impossible to pre-establish the goal and value 
of the data, as the exact goal of these systems is to uncover new and 
unforeseen data purposes. 


While data regulations emphasise the data minimisation principle, 
the practice is that organisations gather as much data as possible. 
The problem in particular for Europe will be that while it could 
potentially prohibit European companies from scraping the web to 
gather training data, it would be certain that American and Chinese 
organisations would do exactly that. For training AI models, enor- 
mous quantities of data are necessary and, in principle, the more data 
they are fed, the more accurate the models get. 


While data regulations emphasise that European organisations 
cannot use systems that have been trained on data that have been 
developed in a way that conflicts with European standards, eg, using 
facial recognition systems that have been trained on photos scraped 
from the internet without the consent of citizens, the practice is that 
organisations buy and use AI systems from American companies 
nonetheless. 
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e While data regulations require that the pre-established goal for 
processing data should be narrowly defined, most contemporary 
organisations employ very general and vague purposes. This also 
means that although the regulations emphasise that data must be 
deleted as soon as the pre-established goal has been attained, the 
practice is that organisations keep data as long as possible, inter alia 
to allow AI programs to train on historical data and to find longitu- 
dinal patterns. 


e While data regulations specify the purpose limitation principle, 
meaning that data can only be used for the pre-established purpose 
for which they are collected, the AI revolution is to a large extent 
based on the idea that data can and should be re-used, that they can 
have a second life and can serve new purposes. 


e While data regulations emphasise the data quality principle, meaning 
that data that are processed should be correct, for example, in order 
to avoid mistakes and decisions based on erroneous information 
having an impact on citizens, LLMs and Al are trained on such large 
data sets that the quality of individual data points is less relevant for 
the accuracy of the model than the quantity of the data on which it 
is trained. 


e While data regulations emphasise control and confidentiality, the 
practice is to share data between organisations, as well as between 
international counterparts, and to make part of the AI models and 
underlying databases available open access to allow for shared crea- 
tivity and learning processes. 


Consequently, many experts have stressed the fundamental incompatibil- 
ity of the data rules, which date back to the 1970s, with contemporary 
data practices. The regulator thus needs to make a choice as to whether 
to maintain the current legal framework, which would mean that many 
of the developments in AI would be prohibited in Europe. Alternatively, 
it could set these principles aside to facilitate the rise of a European AI 
industry, but that would effectively mean abrogating the fundamen- 
tal rights to privacy and data protection, which in turn also contain 
important safeguards against other human rights violations, such as the 
prohibition of non-discrimination. Finally, it could opt for keeping the 
current approach, which is to officially emphasise the various data prin- 
ciples, while turning a blind eye to actual data practice. 

To make matters more complex, there are not only tensions between 
the legal framework and data practice. Alongside setting limits on 
processing and transferring data, the EU’s legal acquis also emphasises 
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informational openness and the need for sharing data. Examples include, 
but are not limited to, the following: 


The same year that the EU adopted the GDPR, it also put out the 
Free Flow of Non-Personal Data Regulation.!'® This regulation 
prohibits data localisation requirements set by states which entail 
that data have to stay within the borders of a country. Alongside data 
localisation requirements, it dissuades restrictions on data mobility 
set by private organisations through legal, contractual and technical 
standards which prevent users of data processing services from port- 
ing their data from one service provider to another or back to their 
own information technology systems. The Regulation emphasises 
that taking away these limitations on the free flow of non-personal 
data is important to facilitate and stimulate the development of new 
technologies such as AI. Where data sets contain both personal and 
non-personal data, in principle, the GDPR should be applied. 7 


The GDPR, the Free Flow of Non-Personal Data Regulation and the 
proposed Data Act emphasise the need for data portability and inter- 
operability of systems: 

o The Free Flow of Non-Personal Data Regulation underlines the 
need for best practices for facilitating the switching of service 
providers and the porting of data in a structured, commonly used 
and machine-readable format, including open standard formats 
where required or requested by the service provider receiving the 
data. 

o The proposed Data Act!!! provides rules on products connected 
through the internet!!? and would be applicable to most HRs and 
AR devices, and, depending on the setting, VR devices. The Act 
contains an obligation to design and manufacture products and 
related services in such a manner that data generated by their use 
are, by default, easily, securely and, where relevant and appropri- 
ate, directly accessible to the user. Where data cannot be directly 
accessed by the user, the data holder must make the data gener- 
ated by their use of a product or related service available to the 
user without undue delay, free of charge and, where applicable, 
continuously and in real time. This shall be done based on a simple 
request through electronic means where technically feasible. The 
user can request to make the data generated using a product or 
related service available to a third party, without undue delay, free 
of charge, of the same quality as is available to the data holder 
and, where applicable, continuously and in real time. Although 
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there are exemptions to both the rules on the access to the data by 
the user and to their freedom to transfer the data to a third party, 
both rights have an important bearing on the danger of commer- 
cial parties becoming too powerful and on lock-in effects. 

The GDPR”® provides that the data subject shall have the right 
to receive the personal data concerning them, which they have 
provided to a controller, in a structured, commonly used and 
machine-readable format, and have the right to transmit those 
data to another controller without hindrance from the controller 
to which the personal data have been provided, if the processing 
of data was based on consent or a contractual relationship (see 
section 5.2.5). When designing synthetic technologies and services, 
developers have a legal obligation to consider this right and to 
make sure that, as far as possible, the technology or service allows 
for the smooth transfer of data to a third party in a way that the 
new party can use the data (the interoperability principle). 


These provisions entail that providers of synthetic technologies cannot 
privatise data and should be conceived as an attempt to break their data 
monopoly (see the discussion in Chapter 4, section 4.4 and section 5.5.5 
below). For example, if a user wants to change to a new HR provider, but 
wants the new HR to be trained on the data the old HR had collected for 
self-learning purposes, this should be made possible by the original HR 
provider. Alternatively, a group of VR players can ask a VR platform to 
transfer their data to another VR platform so that they can keep their 
avatars, their avatars’ history and characteristics. 


The Data Governance Act re-emphasises the need for a free flow of 
data and does so in particular through two mechanisms: 


ie) 


It sets up data intermediaries, in certain other jurisdictions also 
known as data trusts, that act as neutral third parties that connect 
individuals and companies with data users. While they may charge 
for facilitating the data sharing between the parties, they cannot 
directly use the data that they intermediate for financial profit (eg, 
by selling them to another company or using them to develop their 
own product based on this data).'?! 

It promotes data altruism,!” which is defined as the consent by 
data subjects to process personal data pertaining to them, or 
permissions of other data holders to allow the use of their non- 
personal data without seeking a reward, for purposes of general 
interest, such as scientific research purposes or improving public 
services. 
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Although the GDPR is best known for setting limits and imposing 
restrictions on the processing of data, that is only one side of the 
coin. In fact, the reason for adopting an EU-wide data protection 
regime was that until the predecessor of the GDPR was adopted 
in 1995, every European country had its own set of rules and regula- 
tions, which were incompatible and imposed conflicting obligations 
on data controllers. By setting one framework for the entire EU, data 
controllers operating in multiple jurisdiction had to abide by only 
one legal instrument and could freely transfer personal data across 
borders within Europe. Still today, the first article of the GDPR 
emphasises the dual nature of the data protection framework: 
This Regulation lays down rules relating to the protection of natural 
persons with regard to the processing of personal data and rules relating 
to the free movement of personal data. This Regulation protects funda- 
mental rights and freedoms of natural persons and in particular their 
right to the protection of personal data. The free movement of personal 
data within the Union shall be neither restricted nor prohibited for 
reasons connected with the protection of natural persons with regard to 
the processing of personal data. 


Finally, it is important to take into account the rules on the re-use of 
public sector information. Western society has been based on the prin- 
ciple of open and transparent government for centuries. The idea is that 
critical citizens and journalists should be able to check decision-making 
processes in order to expose potential problems and abuse of power. This 
principle also allows historians and scientists to examine archives in order 
to describe and verify how governments operated in certain periods. An 
open government is considered quintessential for a vital democracy. Four 
important developments have taken place in recent years: 


Government documents used to be available in archives, libraries or 
specially designated information centres. Nowadays, more and more 
documents are made available online. This has an important effect on 
what is called ‘practical obscurity’. The fact that in the past one had 
to make the effort to go to the place where the documents were stored, 
request them and view them meant that, in practice, only a limited 
number of people were able to access the information. Broadly speak- 
ing, these were journalists, historians, critical citizens closely following 
the government and lay historians researching their family trees. By 
making the documents public on the internet and not setting any access 
barriers, anyone can now view these documents with ease. 


In the pre-digital age, most documents were ‘passively disclosed’; citi- 
zens, journalists and others were given access to specific documents 
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upon request. They already had to have a rough idea of what they 
were looking for, the disclosure of documents required their initia- 
tive, and the documents were usually made available for a certain 
period of time only. Currently, documents are increasingly disclosed 
actively; the government publishes documents not upon request, but 
on its own initiative. This means that there is no longer a specific 
reason for which a document is made available. Anyone may access 
them and they are made available permanently. 


e The technical possibilities of searching through such documents have 
increased considerably. These include algorithms and AI tools that 
can analyse texts for words, correlations and topics. Whereas previ- 
ously it was primarily individuals who sought access to government 
documents, currently tech companies are best placed to scan and 
analyse the millions of governmental documents that appear on the 
internet every year. 


e The EU has encouraged the Member States not only to make data 
available to further open, transparent and accountable government, 
but also to facilitate the re-use of government data. The idea is that 
governments are sitting on ‘a mountain of data’, while its economic 
potential is not being exploited. Already in 2000, the total value of the 
European Public Sector Information (PSI) was estimated to be around 
€68 billion annually.’? The data are ‘only’ used for furthering public 
interests, it was found, while if the data were released for the commer- 
cial re-use, this would incentivise the data economy. Subsequently, the 
EU adopted a Directive on the re-use of public sector information 
in 2003, which,! following amendments in 20131% and 2019, has 
become even more adamant that governments actively release public 
sector information to enable re-use by commercial parties. 


It is clear that, as with the tensions between data regulations and data 
practice noted above, there are many tensions between the 2019 Open 
Data Directive and the GDPR. The purpose limitation principle in prin- 
ciple prohibits the re-use of data for different purposes, while this is the 
exact goal of the Open Data Directive. In contrast to the data minimisa- 
tion principle, the Directive emphasises the need for storing and making 
available data. While the GDPR underlines data confidentiality and 
security, the Open Data Directive incentivises parties to publish data sets 
freely available online. And while the GDPR emphasises that data may 
only be transferred from one data controller to another data controller 
if both have a legitimate ground for processing, this cannot be verified if 
data are put online. 
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As often happens when there is a complicated interplay between two 
legal instruments of the EU, the EU legislator does not make a choice, 
but leaves the exact relationship between these two instruments in the 
middle.” Although this has led to the practice of European countries 
trying to balance the two instruments, recently, the two European courts 
and in particular the CJEU have made clear that in principle, the GDPR 
and the right to privacy prevail." Still, many countries are unsure how 
the two approaches that run through the EU’s legal acquis should be 
unified and many governmental organisations on which open access obli- 
gations are imposed are uncertain as to what is required of them. For a 
long time, the suggested solution was to anonymise data before putting 
them online, so that the GDPR no longer applies, but as will explained 
in section 5.4.5, this approach may no longer be feasible. 


5.4.5. Data Regulation 


Through the various data and technology laws discussed in this chap- 
ter, the EU aims at laying down a detailed and comprehensive legislative 
package for the twenty-first century. Each of these instruments contains 
valuable provisions, prohibitions, and rights, meaning that taken sepa- 
rately, their introduction should be welcomed. However, there are two 
fundamental questions that emerge, one concerning legal consistency 
and the other whether the underlying philosophy of those legal frame- 
works is valid in an Al-mediated world. 1? 

As to the first point, it is remarkable how little EU legal consistency 
there is in this area. There are several examples of this: 


¢ Ever since the EU started to adopt laws that moved away from the socio- 
economic realm and entered the field of human rights law, little effort 
went to harmonising these with the more established ECHR of the CoE 
and the jurisprudence of the ECtHR. Often, EU law simply mentions 
that account should be taken of the case law of the ECtHR on, for exam- 
ple, the concepts of necessity and proportionality, while leaving it open 
as to what that means exactly for the interpretation of EU laws and legal 
principles. This is important because the EU’s legal corpus, including the 
CJEU judgments, is not on all points consistent with the approach taken 
within the CoE. Examples include, but are not limited to, the following: 


o The difference between the protection of privacy under Article 8 
ECHR and the EU’s data protection regime under the GDPR and 
the Law Enforcement Directive. "0 
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o The differences between the prevention of discrimination under 
Article 14 ECHR and the EU laws on specific forms of discrim- 
ination, such as on the grounds of race and ethnic origin,'! 
discrimination at work on the grounds of religion or belief, disa- 
bility, age or sexual orientation,' equal treatment for men and 
women in matters of employment and occupation,'* equal treat- 
ment for men and women in the access to and supply of goods 
and services,!* and discrimination based on age, disability, sexual 
orientation and religion or belief beyond the workplace. 

o The difference between the EU’s approach to liability of internet 
intermediaries, focussing on safe harbours and a notice and take- 
down or notice and action regime, and the ECtHR’s focus on the 
freedom of expression and the obligations of publishers. 


Because of the discrepancy between both legislative corpuses, it matters 
for the outcome of a legal dispute whether it is treated under EU law or 
the ECHR and whether it is judged by the CJEU or the ECtHR. 


e The EU adopts so much legislation and in such broad terms that 
it will be almost impossible for national legislators to bring their 
full legislative corpus into compliance with EU law. EU laws, such 
as the GDPR, prevail over the national laws of Member States. But 
almost every law will entail some form of data processing, such as 
when referring to the requirement to keep or produce ‘documents’, 
‘files’, ‘registers’ or ‘information’, and virtually all of the specific 
documents, files, registers or information will contain personal 
data. No Member State has assessed its entire legislative corpus and 
revised it in full to bring it into conformity with the GDPR; rather, 
Member States have chosen to update a handful of laws central to 
data processing practices and have stressed that all other laws must 
be interpreted ‘in light of the GDPR’. This applies to the various EU 
laws on non-personal and other types of data as well. In addition, 
as discussed above in section 5.4.4, it often adopts contrasting legal 
regimes, without making clear how Member States should apply 
these in practice. 


e The EU is often not consistent or harmonious in its approach and 
terminology: 


o The GDPR applies different levels of protection to personal data, 
sensitive data, and anonymous and aggregated data, places pseu- 
donymous data somewhere in between anonymous and personal 
data, and recognises several types of sensitive data, such as genetic 
data, biometric data and data concerning health. Many of the 
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proposed Acts now on the table use different terminologies. The 
e-Privacy Regulation distinguishes between metadata, including 
location and traffic data, electronic communications data, elec- 
tronic communications content and electronic communications 
metadata, and also makes reference to publicly available directo- 
ries with data about end-users. The AI Act defines and regulates 
still different types of data, such as training data, validation data, 
testing data and input data. The Digital Markets Act (DMA) 
(which is discussed in more detail in section 5.5.5) emphasises 
the difference between aggregated and non-aggregated data, and 
between personal and anonymised data, but also refers to data, 
both in contrast to the definition of personal data, for which refer- 
ence is made to the GDPR, and to that of non-personal data, for 
which reference is made to the Regulation on the transfer of non- 
personal data. The Regulation on the transfer of non-personal data 
itself does not give a definition of non-personal data, but of 
data, which is seen as encompassing all data other than personal 
data. The Digital Services Act (DSA) (which is discussed in more 
detail in section 5.5.4) refers to illegal content as a special category 
of data; the Data Governance Act, like the DMA, distinguishes 
between three types of data, though not between data, personal 
data and non-personal data, but between data, non-personal 
data and metadata; the proposed Data Act only refers to data; 
and the Open Data Directive refers to dynamic data, research data 
and high-value data sets as categories of data that are specifically 
regulated. How these various categories of data and the partial 
overlaps and contrasts that exist between them will have to be 
interpreted is left open. 

If these different and sometimes conflicting categorisations of 
data and the various regimes for protection that are connected 
to them are not already difficult enough from a compliance 
perspective, parties that want to abide by the various regimes 
that may be applicable to them are themselves categorised 
differently in each legal regime, with different roles and respon- 
sibilities being connected to these categories. The GDPR 
differentiates between the data subject, the data processor 
and the data controller, the Regulation on the transfer of non- 
personal data speaks of service providers, users and professional 
users, the DSA refers to information society services, recipi- 
ents of services, consumers, traders, intermediary services and 
online platforms, the DMA differentiates between gatekeepers, 
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core platform services, cloud computing services, software 
application stores, online intermediation services, online search 
engines, ancillary services, online social networking services, 
identification services, video-sharing platform services, number- 
independent interpersonal communication services, operating 
systems, end-users, business users and undertakings, the Data 
Governance Act makes reference to data holders and data users, 
the proposed Data Act refers to users, data holders, data recipi- 
ents and data processing services, and the AI Act, to give a final 
example, has rules for providers, small-scale providers, users, 
importers, distributors and operators. 


This means that the same entity may be qualified differently under 
one legislative regime compared to another, the same data may be 
categorised differently under one act than under another and, conse- 
quently, the types of rules, prohibitions and obligations applicable to 
data processing activities may vary significantly or even conflict on 
certain points. 


In addition, besides the matter of legal consistency, and perhaps 
more fundamentally, AI and the underlying data technology revolu- 
tion question the groundwork of European data regulation, which is 
to differentiate between different data categories and attach different 
levels of regulatory protection to it. For example, the GDPR differ- 
entiates between non-personal data (which are not protected) and 
personal data (which are protected) and sensitive personal data (the 
processing of which is, although there are many exceptions, in princi- 
ple forbidden).'8° Another well-known distinction is between content 
communications data, which falls under the right to informational 
secrecy, and meta-communications data, such as data about who has 
communicated, when and where, which traditionally fall outside the 
scope of that right, as they would not reveal what was communicated. 
The idea of having several strictly separated types of data, each with 
their own scope of protection, is increasingly criticised for reasons such 
as, but not limited to, the following: 


¢ Working with well-defined and delimited definitions of different 
types of data only works if the status of data is relatively stable and 
falls into one category or the other for a longer period of time, while 
in the data driven environment, this is less and less so. The nature 
of the data in AI processes is not stable, but can be highly volatile. 
A data set containing ordinary personal data can be linked to and 
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enriched with another data set so as to derive sensitive data; the data 
can then be aggregated or stripped of identifiers and be turned into 
non-personal, aggregated or anonymous data; subsequently, the data 
can be de-anonymised or integrated into another data set in order to 
create personal data. All this can happen in a split second. The ques- 
tion is therefore whether it makes sense to work with well-defined 
data categories if the same ‘datum’ or data set can literally fall into a 
different legal category from one second to the next. 


It is increasingly difficult to determine the status of data precisely. As 
the Working Party 29 put it, ‘the assessment of whether the data allow 
identification of an individual, and whether the information can be 
considered as anonymous or not depends on the circumstances, and 
a case-by-case analysis should be carried out with particular refer- 
ence to the extent that the means are likely reasonably to be used 
for identification’.'’ The GDPR holds that in order to determine 
whether a datum is to be considered ‘personal’, account should be 
had of the means that can reasonably be expected to be used for iden- 
tification. Therefore, in order to determine the current status of a 
datum or data set, the expected future status of the data must be 
taken into account. Given the democratisation of technologies and 
the minimal investment required, it is increasingly likely that when a 
database is shared or otherwise made available, there will be a party 
that will combine it with other data, enrich it with data scraped from 
the internet or merge it into an existing data set. Thus, it is increas- 
ingly likely that if an anonymised data set is made public, there will 
be a party that will de-anonymise it or combine it with other data to 
create personal profiles, that if a set of personal data is shared, there 
will be a party that will use that data to create a data set containing 
sensitive personal data, and so on. On the other hand, there will be 
other parties that have access to that data, but that will not engage 
in such activities; parties who will not use the data, use it as it is 
provided, or even de-identify a database containing personal data. 
Who will do what is not clear in advance. The legal category to which 
the data belongs is therefore no longer a quality of the data itself, but 
a product of a data controller’s efforts and investments. 


The question is whether the distinction made between different cate- 
gories of data is still relevant. The underlying rationale is that the 
processing of personal data has an effect on natural persons, while 
the processing of non-personal data does not, and that the process- 
ing of sensitive personal data may have very significant consequences 
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(greater than the processing of ‘ordinary’ personal data), so that 
the latter are subject to the most stringent regime, personal data fall 
under the ‘normal’ protection regime, and the processing of non- 
personal data is not subject to any restrictions. Pseudonymisation 
does not ensure the full protection of individuals, but it does greatly 
reduce the number of people and organisations that can link data to 
specific individuals, which is why pseudonymous data are put a legal 
purgatory, somewhere between non-personal and personal data. The 
question is to what extent this rationale is still tenable in the twenty- 
first century. Data processing on the basis of aggregated data, for 
example, can also have very big individual and social consequences. 
Many intelligence agencies prefer meta over content data, not only 
because they are easier to analyse, but also because they reveal more 
about a target. Processing of ‘ordinary’ personal data can often be 
regarded as very sensitive, such as with processing data about chil- 
dren, though such data are not provided protection under the strictest 
legal regime afforded by the GDPR. 


Although these questions have plagued Europe’s approach to data regu- 
lation for years, Big Data, AI and the synthetic technologies discussed in 
this book have amplified the existing concerns. Consequently, the regu- 
lator needs to determine to what extent it can hold on to the current 
approach and what alternatives exist. Should, for example, all data 
simply be treated as personal data, so that one regulatory regime applies 
or would that stifle innovation too much? 


5.4.6. Conclusion 


This section shed light on how current and future legal regimes on tech- 
nology and data will affect synthetic technologies. As with the GDPR 
(see section 5.2), the application of the rules discussed in this section 
to synthetic technologies can be divided into three categories. First, 
there is the category of rules that set clear boundaries, limitations and 
conditions for the production, servicing and use of these technologies. 
Examples include the prohibited AI systems in the AI Act, the cyberse- 
curity requirements in the Cyber Security Act and the Cyber Resilience 
Act, and the rules on interoperability and device integrity. Second, there 
is the category of rules for which it is unclear how they would apply 
to synthetic technologies. This can be either because this is dependent 
on a case-by-case analysis or because the legal frameworks are unclear 
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or ambiguous. Examples include the general rules on product safety 
and indirect references to psychological health, the non-legally binding 
rules on trustworthy AI, the procedural safeguards contained in the AI 
Act and the rules on technology and data export. In addition, there are 
several frameworks with opposing obligations, meaning that it can be 
unclear to organisations developing or using synthetic technologies how 
the various rules can be respected at the same time. Third, there is the 
category of rules that may need revision, such as the sharp distinction 
under the current legal framework between non-personal and personal 
data, the relative absence of rules on the import of data and technology, 
and the choice for risk-based regulation, in particular under the AI Act. 


5.5. LIABILITY AND PROCEDURAL LAW 
5.5.1. Introduction 


This section discusses matters concerning law enforcement, liabil- 
ity, procedural law and consumer and anti-trust law. First, it assesses 
what effects synthetic media may have in the courtroom (section 5.5.2). 
Second, the difficulty of law enforcement and the distribution of respon- 
sibilities regarding synthetic technologies is discussed, in particular by 
homing in on the difference between ex ante and ex post regulation 
(section 5.5.3). The liability of professional parties for AI products and 
distributing synthetic media (section 5.5.4), and the rules and limits set 
out in consumer and competition law (section 5.5.5) are then covered. 
Finally, a small wrap-up is given (section 5.5.6). 


5.5.2. Procedural Law 


Because it is the ECtHR’s task to ascertain whether the proceedings 
at the national level are fair, including the way in which evidence is 
evaluated and used, it will establish whether the evidence is presented 
in such a way that it can guarantee a fair trial.!585 While Article 6 § 
1 does not explicitly guarantee the right to have witnesses or experts 
called, the ECtHR has ruled that when domestic courts refuse to do so, 
they must give sufficient motivation, such refusal must be non-arbitrary 
and a refusal will conflict with the right to a fair trial if it amounts to 
a disproportionate restriction of the litigant’s ability to present argu- 
ments in support of their case. Importantly, the Court has underlined 
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that a difference of treatment in respect of the hearing of the parties’ 
witnesses may undermine the equality of arms. 

Equality of arms implies that each party must be afforded a reason- 
able and equal opportunity to present their case, as well as the evidence 
for their case. When opposing parties have different opportunities with 
regard to access to relevant information or if governments present 
evidence in a summary way, disallowing citizens to adequately challenge 
the evidence, this principle may be violated. The refusal to allow an alter- 
native expert examination of material evidence can result in a violation 
of the right to fair trial, in particular if it is hard to challenge a report by 
an expert without the assistance of another expert in the relevant field. 
A failure of the prosecution to disclose relevant technical details — for 
example, those details on which an expert report is based — may breach 
equality of arms as well as the right to an adversarial hearing. This also 
holds true for non-disclosure of evidence to the defence. In criminal 
law, the presumption of innocence requires, inter alia, that the burden 
of proof is on the prosecution and that any doubt should benefit the 
accused (in dubio pro reo). When the national court’s reasoning leading 
up to a conviction is not well reasoned, this can be seen to be in conflict 
with the right to a fair trial. A final important aspect of the right to a fair 
trial is the right to effective participation in the proceedings. 

The ECtHR has also found that there are procedural requirements 
implicit in the substantive rights contained in the ECHR. This means 
that when an administrative body or court of law takes a decision that 
affects a substantive right (eg, the right to privacy, the freedom of expres- 
sion or the right to property), these procedural requirements must be 
respected. For example, under the right to property (Article 1 of the First 
Protocol to the Convention),'® persons affected by a measure interfer- 
ing with their possessions must be afforded a reasonable opportunity 
to put their case to the responsible authorities for the purpose of effec- 
tively challenging those measures. Even in situations where no individual 
procedural deficiencies in themselves decisively affect the proportionality 
of a forfeiture measure, but, taken together, the relevant factors result in 
uncertainty and imprecision, they may render a forfeiture disproportion- 
ate to the legitimate aim pursued. 

Under Article 8 ECHR, the ECtHR has adopted a wide range of 
procedural requirements.'*! Persons whose right to privacy is affected 
by a decision by a governmental body should have the ability to bring 
the matter before the national courts, they should be involved in the 
decision-making procedure to the degree sufficient to provide them with 
the requisite protection of their interest, and they should have adequate 
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access to reports and documents that are relevant to the proceedings. 
The ECtHR has found that it is the responsibility of authorities to inves- 
tigate the reliability of purported evidence relied on by authorities or 
to litigate to obtain such information, from an early stage and as soon 
as practically possible, and provide for the material to be viewed. The 
ECtHR has also found that decision-making procedures should be swift, 
fair and independent. Citizens must be heard and have influence over the 
decision whenever possible. The decision-making process must be timely 
and the persons affected must have a right to challenge assertions, must 
be allowed to request second opinions and must have a right to legal 
representation. The decisions must be comprehensible and fair, and there 
must be a way to challenge the decisions before another legal body.!? 

There are several questions as to how to apply the current doctrine of 
procedural law and rules with respect to evidence to synthetic technolo- 
gies, such as, but not limited to: 


e When a HR causes harm or makes a mistake, it is the company operat- 
ing the HR that would most likely have access to the relevant data. There 
is a real danger that these data might not be disclosed by companies, 
citing reasons of business secrets, confidentiality or privacy protection. 
Because many of these companies are based in different jurisdictions, 
it may be difficult to obtain a court order to hand over all the relevant 
data (see however the AI Liability Directive discussed in section 5.5.4). 
A natural person or company who owns or leases the HR may not be 
able to demonstrate that it was not their command or sensory input 
that caused the harm. In addition, the terms and conditions of the 
HRs may exonerate the company marketing the HR from liability for 
damage and harm, which may give rise to complicated questions over 
the validity of these terms and conditions. 


e When harm is caused by a synthetic technology, a myriad of parties 
are often involved, and these parties may be based in multiple juris- 
dictions, with different rules of procedure, standards for evidence 
and obligations for parties to testify. When a HR is designed in the 
Netherlands, built in China, marketed by a US-based company and 
used by a person in Switzerland, this may trigger a complex web 
of rules and procedures. Similarly, when one VR player does some- 
thing harmful to another VR player and the first is based in Ghana, 
the second is based in Germany and the platform itself is based in 
Taiwan, intricate jurisdictional problems arise. 1# 


e It may not always be clear how HRs or Al-driven avatars operate 
and respond to input as a self-learning algorithm may develop in 
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ways that are not necessarily understandable to humans. If there is a 
legal dispute between a user or victim of an Al-driven creature and a 
provider of synthetic technologies, a clear question as to the inequal- 
ity of arms may arise, as citizens do not have access to the data or 
the underlying technology and do not generally have sufficient tech- 
nological knowledge to understand information once it is disclosed. 
This might require them to hire experts and witnesses to interpret the 
data, which could amount to substantial costs. Because many legal 
disputes involve an individual citizen fighting a large tech company 
which, besides technical expertise, usually has access to a large legal 
team and sufficient recourse to sit out a court case for years, there will 
be a substantial inequality in court cases. 


e In VR worlds in particular, there is a question as to what extent normal 
courts hold sway over these platforms, which may differ between juris- 
dictions, because VR worlds often adopt their own forms of dispute 
resolution. If conflicts between different players arise, they can be 
required to first use the platform’s dispute resolution mechanism, 
as well as if they have a complaint about the platform itself, either 
because their avatar is blocked or removed, or because they think the 
platform is liable for the behaviour of other avatars — for example, 
because it condones or even actively promotes violent or inappropri- 
ate conduct. These private forms of dispute resolution may be biased 
in favour of the VR platform and may not conform to the rules of 
procedure applicable to courts. 


e Itis clear that if a governmental authority takes a decision that signif- 
icantly affects a person, such as placing a HR care robot in the home 
of a person and subsequently takes it away, all procedural safeguards 
must be adhered to, such as informing the person in full, consulting 
them, and taking due account of their views and potential objec- 
tions. In principle, such rules, as established under the ECHR, do 
not directly apply to private parties, although there may be a positive 
obligation for states to ensure that private parties respect the human 
rights of citizens. This could be yet another motive for states to priva- 
tise the execution of public tasks, such as medical care. 


e The use of ChatGPT and other LLMs by both judges and lawyers has 
raised concerns. First, private companies are given access to prelimi- 
nary pleas and judgments, which would put them in a position to 
use or abuse that knowledge. Second, as explained in Chapter 4, 
LLMs are not neutral; they are biased. This would mean that algo- 
rithmic biases would have a potential impact on the outcome of 
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court cases. Third, LLMs make mistakes and fabricate facts, sources 

and evidence. There are already examples of how their use can lead 

to problematic results: 
[One case] involved a man suing an airline over an alleged personal injury. 
His legal team submitted a brief that cited several previous court cases in 
an attempt to prove, using precedent, why the case should move forward. 
But the airline’s lawyers later wrote to the judge to say they could not find 
several of the cases that were referenced in the brief. ‘Six of the submitted 
cases appear to be bogus judicial decisions with bogus quotes and bogus 
internal citations’, Judge Castel wrote in an order demanding the man’s 
legal team explain itself. Over the course of several filings, it emerged 
that the research had not been prepared by Peter LoDuca, the lawyer 
for the plaintiff, but by a colleague of his at the same law firm. Steven 
A Schwartz, who has been an attorney for more than 30 years, used 
ChatGPT to look for similar previous cases. In his written statement, 
Mr Schwartz clarified that Mr LoDuca had not been part of the research 
and had no knowledge of how it had been carried out. Mr Schwartz 
added that he ‘greatly regrets’ relying on the chatbot, which he said he 
had never used for legal research before and was ‘unaware that its content 
could be false’. He has vowed to never use AI to ‘supplement’ his legal 


research in future ‘without absolute verification of its authenticity’.!4 


As with universities, which need to establish for what tasks students 
can use LLMs (eg, they may be prohibited from using such models 
for writing an assignment directly, but potentially they can be allowed 
to ask a LLM to identify the most relevant literature on the topic of 
a writing assignment, to make a list of pros and cons for a certain 
thesis, to remove grammatical errors in the text or to act as a teacher, 
giving feedback on a draft assignment), the legal branch will need 
to set rules for the use of LLMs by lawyers, public prosecutors and 
judges. 

The rise of Generative AI may further the push for legal automation. 1*5 
Although potentially advantageous in terms of efficiency, speed, legal 
consistency and explicability of the legal process, there are many 
issues discussed in this book that give rise to caution. In addition, 
there are two elements important for the legitimacy of a legal system 
as a whole and judgments in particular. On the one hand, a court 
should establish the right, most logical and consistent interpreta- 
tion of the law and the right, most logical and consistent application 
of that law to a specific set of circumstances. Although infinitely 
complex, AI could potentially help to deliver on this point. On the 
other hand, the legitimacy of a judgment depends on the process, 
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on each party having the feeling that they have been seen, that their 
arguments have been heard and that their worries are taken seriously 
by another person (see Chapter 4, section 4.3). This point would be 
undermined when parts of the legal process would be automated, let 
alone when human judges would be substituted by robo-judges. 


DFs, in particular, may challenge a number of rules and presumptions 


under current procedural law. These include the following: 
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If material is not released immediately, but a video is ‘archived’ and 
only released after someone’s death (eg, a compromising video to 
damage their moral legacy), who can dispute the authenticity of a 
video and on what grounds? 


DF technology may lead to truly authentic material being disregarded 
because it cannot be established beyond reasonable doubt that such 
material is authentic; a suspect can also always claim that a video, 
image or audio clip is fake, even after a court has delivered a final 
sentence. 


Judges will typically assume evidence to be authentic unless there 
are contra-indications. This means that it either has to be appar- 
ent from the evidence submitted that it has been tampered with 
or that one of the parties has to put forward that the evidence is 
inauthentic. The former will become ever less likely because DF 
technology will become ever more powerful. The latter will not 
always be possible. For example, there are criminal cases in which a 
person is convicted in abstentia. In addition, there are people with 
mental limitations who might either not have the capacity to under- 
stand that evidence has been tampered with (for example, children 
or persons with an extremely low IQ) or do not remember what 
happened (for example, people suffering from dementia or people 
who were intoxicated). 


Considering the prediction that in a few years’ time, more than 90 
per cent of all online content will be manipulated, it could be argued 
that all evidence in court should be assessed on authenticity. This, 
however, would involve considerable cost. It is often far from clear 
whether an audio clip, a video, a picture, satellite data or a tweet is 
authentic or not and how that should be established. AI programs can 
filter out about half of DFs, but not all and, as discussed previously, 
some of those programs only give an estimated likelihood of authen- 
ticity, such as that there is a 71.2 per cent chance that the material 
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is authentic. This leaves courts with the task of making particularly 
complex evaluations, for example when the main piece of evidence in 
a criminal case is a video, the authenticity of which is estimated to 
be 70 per cent by the Al-detection program. Would this percentage 
be enough to arrive at a conviction and where should the boundary 
be drawn? Would there be a trend towards reliance on circumstantial 
evidence? Should there be a difference in the required authenticity 
percentage for administrative law, civil law and criminal law cases? 
What if, in a custody case, both parties produce a video of the other 
parent being abusive towards their children aged one and two, but 
both videos have an authenticity percentage of 50 per cent, with the 
neighbours saying that they clearly heard someone screaming, but 
could not be more precise as to who? Should custody be awarded to 
both, one of them or neither? What happens if both parties hire their 
own experts to test the authenticity of the material and these experts 
arrive at different conclusions? Would this result in an even bigger 
role of experts in legal proceedings and an even longer legal process? 
Do judges have sufficient expertise to assess the quality of the expert 
witnesses and their statements? 


e Next to the various questions over the use of digital evidence when 
reaching a verdict, there will be questions over the admissibility 
of evidence in court. Like with the former questions, under the 
current legal paradigm, it is not always clear which party must 
meet what bar at what point in the legal process for evidence to 
be admissible. 


5.5.3. Ex Ante and Ex Post Regulation 


Ex post regulation is regulation that focuses on the use of a technology 
and assesses whether that use conforms with the legislative framework 
only after that use has taken place. Ex ante regulation can take the form 
either of bans of or conditions on the production of technology, or bans 
of or conditions on putting technologies on the market. An intermedi- 
ate regulatory option is assessing the legitimacy of intended use of a 
technology before that technology is put into practice and assessing 
the legitimacy of content produced through that technology before it is 
made public. The earlier in the process regulation is set, usually the more 
impact such regulation has. 
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Figure 5.4 Regulatory options in the value chain, from production of 
technology to assessing ex post usage 


The various legal instruments setting product requirements (see 
section 5.4.2) and the AI Act (see section 5.4.3) would, if adopted, 
provide bans on a limited number of synthetic technologies. Most of 
them concern specific use cases, such as the use of ‘real-time’ remote 
biometric identification systems in publicly accessible spaces for the 
purpose of law enforcement. The Dual Use Regulation will set limits to 
the type of technologies that can be exported to countries outside the EU. 
In addition, some European countries have set restrictions on market- 
ing certain technologies, such as spyware. Finally, there are obligations 
(mostly self-regulatory) for internet providers to filter out child pornog- 
raphy from their services." 

The GDPR requires a Data Protection Impact Assessment for data 
processing operations that are likely to cause harm, and the AI Act 
requires such assessments for high-risk AI systems. However, these have a 
relatively limited impact on synthetic technologies. The GDPR, for exam- 
ple, gives three situations in which Data Protection Impact Assessment 
needs to be executed in particular, namely: 


e in the case of a systematic and extensive evaluation of personal 
aspects relating to natural persons, which is based on automated 
processing, including profiling, and on which decisions are based 
that produce legal effects concerning the natural person or similarly 
significantly affect the natural person; 
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e in the case of large-scale processing of special sensitive data; and 


e inthe case of large-scale systematic monitoring of a publicly acces- 
sible area. 


Impact assessments generally concern the specific use of a technology 
by one data controller for one specific goal. It is questionable whether 
any of the use cases of synthetic technologies would fall under this obli- 
gation. Most AR users, for example, do not systematically monitor the 
public sphere, but hop in and out of the public, private, professional and 
semi-private spheres randomly. A DF pornographic video of someone 
does not process sensitive data on a large scale, although perhaps porn 
sites could fall under this provision. A medical HR might carry out a 
systematic and extensive evaluation of one person, but usually not of 
more than one, while the GDPR speaks of ‘persons’ plural. Potentially, 
when a HR is used in medical facilities to evaluate multiple patients, 
this could fall within the scope of the obligation to carry out a Data 
Protection Impact Assessment. Consequently, there are some doctrines 
that provide or will provide ex ante rules and some that provide rules that 
fall in the intermediate category. 

Next to these legal standards, the introduction of AI has had the effect 
that companies have set controls earlier in the value chain. Traditional 
content production technologies, such as Microsoft Word or Paint, do 
not set limits on which content you are allowed to create. You could, for 
example, write a Nazi pamphlet or draw naked pictures of minors using 
these technologies. The first threshold is generally on the distribution 
of such material imposed by service providers, who ban illegal material 
through various mechanisms. Although there are dark web alternatives, 
most popular AI programs have many inbuilt safeguards that prevent 
people from using the program to pursue illegal activities or producing 
unlawful content. It is unclear how long these strict safeguards will be in 
place, as many have urged that these are overly broad, stifle creativity and 
limited legitimate use cases. 

However, apart from these, the current regulatory framework is almost 
exclusively based on ex post regulation. This means that, in general, 
technologies can be produced, traded and used without governmental 
intervention. Law enforcement authorities and DPAs will primarily focus 
on the use of the technologies in the specific circumstances of the case and 
will assess whether that use has violated one of the principles contained 
in criminal law, tort law, the human rights framework or otherwise. 
Outside the domain of data protection and, in particular, with respect to 
civil law and administrative law cases, however, there is no governmental 
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organisation that monitors the respect for the legal instruments in place. 
This means that uses of technologies will usually only be scrutinised by 
a court after a victim has issued a complaint. Although the GDPR does 
allow countries to facilitate collective actions, these are seldom used in 
practice. In Europe, there is no legal culture comparable to the flourishing 
practice of class actions in the US. 

There are arguments to be made for each type of regulation. Ex post 
regulation allows scientists, commercial parties and citizens to freely 
develop and experiment with technologies, and thus incentivises inno- 
vation. Experimentation may bring to light new and unforeseen ways 
in which technologies can be used. Ex ante regulation, on the contrary, 
may have a stifling effect; this form of regulation disallows legitimate 
uses of technology. Banning a technology that is abused in 99 per cent of 
use cases also disallows the 1 per cent of legitimate use cases and limits 
finding new use cases through experimentation. In addition, it may be 
difficult to enforce ex ante rules — for example, because technologies are 
produced outside Europe and software can be downloaded with a click 
by users in Europe from a website hosted anywhere. Restrictions on sell- 
ing or offering technologies may also result in a black market. 

For regulatory controls in the middle of the value chain, significant 
time and resources are needed, such as when assessing whether the 
intended use of a technology or the intended publication of content is 
legitimate or not. This assessment requires answers to a range of difficult 
questions regarding the interpretation of legal concepts, the applicability 
of these concepts to the case at hand and the legitimacy of the content, 
without it always being clear whether harm would be done by intended 
use or publication. An additional question would be who should under- 
take these assessments. Governmental authorities are ill-equipped to 
assess relevant uses of synthetic technologies and the publication of 
synthetic material. However, leaving such evaluations to private parties 
means that commercial interests will affect these decisions and that there 
is no democratic control over important regulatory standards. In addi- 
tion, given that most tech firms active in Europe are based in the US, 
they may need to confirm to the American approach to human rights, 
which is different from the European approach (eg, a bigger emphasis 
on the freedom of expression and less emphasis on the right to privacy 
than in the European approach). Finally, one way in which private parties 
tend to approach content management is through using algorithms. This 
may lead to a rigid and black-and-white interpretation of legal concepts, 
and could conflict with the prohibition on automated decision making 
contained in the GDPR. 
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The current regulatory focus on ex post regulation, in turn, has an 
important consequence. It is arguably the cause of the biggest legal and 
regulatory bottleneck in the digital domain: the enforcement of the regu- 
latory framework. Ex post regulation entails that the legitimacy of the 
use of technology in practice as well as the collection, processing and 
publication of information will be assessed retrospectively. As explained 
in Chapter 2, there is a myriad of synthetic technologies that are and will 
be used for a wide variety of use cases, and they allow for the processing 
of data on an unprecedented scale. The enormous amount of synthetic 
media published every day means that adequate ex post control over the 
use of technology and the publication of content by governmental organ- 
isations is almost impossible. They can only realistically assess a fraction 
of that content and generally do so only after a complaint has been made 
or if the content has otherwise been brought to their attention. 

This means that governmental organisations are likely to focus on 
a handful of more extreme violations of the law (eg, related to bodily 
privacy or poignant forms of discrimination) and that more compli- 
cated legal matters (eg, where both the right to privacy and the freedom 
of expression come into play (see section 5.3.5) or where different 
parties hold different property rights (see section 5.3.2) are mostly left 
untouched. This ensures a normalisation of minor violations of law. 
The focus on ex post regulation will mean that a violation will already 
have occurred; the damage is already done. Recovering damages is often 
difficult and if a court case is successful, the damages awarded are often 
quite low. A victim of revenge porn, for example, often receives several 
hundred euros, while the publication of the image can have an impact 
on them for the rest of their lives. In addition, by going to court or when 
governmental organisations act, more focus and attention may be put 
on the alleged illegal content (the so-called ‘Barbra Streisand effect’). '8 

From the focus on ex post regulation and the application of DPIAs 
to specific usage of a technology by an individual data controller flows 
a focus under the dominant current legal paradigm on individual rather 
than general interests. If a particular digital technology has an unde- 
sirable social impact, then ex ante regulation is the obvious choice. For 
example, if the EU were to believe that DF technology as such is harm- 
ful to society because it contributes to the post-truth era and because a 
rough estimate of 96 per cent of all DFs involve non-consensual pornog- 
raphy, then a full or partial ban could be a realistic choice. By contrast, 
in ex post regulation, a specific application is assessed. Thus, the focus 
is primarily on the private interests of one or a few legal subjects. For 
example, the person portrayed in a specific DF porn video will be the 
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subject of focus rather than the impact of the technology as such on the 
societal position of women. 

The emphasis on the protection of individual interests works well 
for many traditional legal disputes: a building permit, a request for 
compensation after a defamatory publication, or an alleged interference 
of privacy by the government, where someone’s phone is tapped for a 
certain period of time. Invariably, the alleged interference is limited to 
a specific person or a small group. It is delimited in time and space, and 
the interest at stake is individual and clearly determinable. It is under- 
standable that the EU, the ECtHR and many European countries have 
extrapolated this approach to the digital domain as it is also suitable for 
many digital incidents. However, there are also issues at play that are 
difficult to address through this approach. With data-driven technolo- 
gies, it can be difficult or impossible to delineate their application in 
time, space and person. For example, the cameras on AR goggles used 
in the public domain do not affect one particular individual specifically; 
they film everyone the user passes. HRs that rely on predictive policing 
and patrol more in certain neighbourhoods than in others affect all those 
living in the neighbourhood. DFs inciting hatred against minorities do 
not affect one or a small number of persons. Many synthetic applica- 
tions are not focused on specific individuals, but affect a multitude of 
individuals, groups and society as a whole. 

Alongside the emphasis on protecting individual interests, there is 
an emphasis under the dominant regulatory approach on individual 
control and subjective rights. It is generally up to citizens themselves to 
defend their interests. This is reinforced by the immense pressure on law 
enforcement authorities and data protection authorities, and their inabil- 
ity to adequately assess the legitimacy of most use cases of synthetic 
technologies and the publication of synthetic content. The focus on indi- 
vidual control, informed consent and informational self-determination 
is grounded in the conception of humans as homo economicus: if citi- 
zens have access to all relevant information and have strong subjective 
rights, in principle, they are in the best position to pursue their own 
interests. Again, there is logic to this approach, and the focus on indi- 
vidual interests and individual rights was a general feature of most 
legislative projects in Western democracies in the 1980s, the 1990s and 
the early 2000s. Gradually, however, a new understanding of individual 
responsibility has emerged as the consequences of ‘neoliberal’ regula- 
tory approaches have become widely recognised. Of course, it is now 
generally accepted, children should learn to make healthy choices, but 
it also helps if the school canteen offers them more than just junk food. 
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Of course, people should use energy sparingly, but global warming will 
not be tackled by citizens using energy-saving light bulbs. Of course, one 
should be alert to specific forms of discrimination, but there is also a 
positive obligation for the government to counter systemic racism. 

If anywhere, the limits of individual control are visible in the digital 
domain. Individuals are often unaware that their data are being collected 
and processed. If they are, they do not always know who is processing 
their data and why; in order to find out, information requests and some- 
times legal procedures are required. On the basis of that information, the 
citizen should assess whether the processing of their data is legitimate, 
for which they need legal, technological and organisational knowledge 
which they often lack. Even if they had such information, assessing all 
data processes that contain their information or otherwise affect them, 
which will easily consist of several thousand projects, is practically 
impossible for reasons of time, energy and resources. If, finally, citizens 
have assessed a data process, do understand it, can demonstrate individ- 
ualisable harm, have come to the conclusion that the use of a synthetic 
technology violated their rights and do go to court, they are usually up 
against governmental organisations or big firms that have the technical 
knowledge, time and resources to fight lengthy legal proceedings. 

Consequently, although there is no perfect solution, an option would 
be to focus more on ex ante rules, prohibitions and conditions rather 
than primarily relying on ex post regulation, as the current regulatory 
framework does. This would not only prevent harm instead of remedy- 
ing it, but would also allow for better oversight and enforcement of legal 
standards, as the earlier in the chain rules are set, the more limited the 
number of actors whose activities need to be scrutinised and thus the 
easier it is for DPAs, law enforcement authorities and other supervisory 
authorities to perform their tasks. 


5.5.4. Liability for Synthetic Technologies and Content 


Another way of reducing the burden on citizens is by placing greater 
duties of care on private parties. There are three important regimes in 
the EU that deal with liability and responsibility of parties. First, there is 
the regime outlined in the proposed AI Liability Directive, which primar- 
ily focuses on producers of AI systems. Second, there is the proposed 
Liability for Defective Products Directive, which pays particular atten- 
tion to smart products and AI.” A third regime concerns the liability 
of internet intermediaries, which are essential for publishing synthetic 
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content and for providing a reservoir of content that can be used by 
synthetic technologies (eg, the content for DFs and the information 
projected on the screens of AR devices). 

While not addressing all of the problems of addressing synthetic tech- 
nologies under current procedural law (see section 5.5.2), some of these 
may be tackled through the proposed AI Liability Directive!’ regarding 
non-contractual liability. The Directive deals in particular with prob- 
lems surrounding burden of proof, access to information and equality of 
arms. The reason for proposing this Directive is that standing national 
liability rules are generally deemed unfit for handling liability claims for 
damage caused by Al-enabled products and services. This is particu- 
larly so because, under the existing doctrines, victims need to prove a 
wrongful action or omission by the person who caused the damage. The 
Directive makes it clear that the specific characteristics of AI, including 
complexity, autonomy and the opacity of these systems (eg, black boxes), 
often make it difficult or prohibitively expensive for victims to identify 
the entity liable and meet the requirements for a successful liability 
claim. Consequently, adopting new liability rules is considered necessary 
in light of synthetic technologies such as HRs. "°! 

The AI Liability Directive, which is linked to the AI Act (see 
section 5.4.3), provides that national courts can be empowered to order 
the disclosure of evidence from those persons upon request.!? This 
can be at the request of either a potential claimant who has previously 
asked a provider, a manufacturer, a distributor, importer or a user of an 
AI system to disclose relevant evidence at its disposal about a specific 
high-risk AI system that is suspected of having caused damage, but was 
refused. In support of that request, the potential claimant must present 
facts and evidence sufficient to support the plausibility of a claim for 
damages. However, the national court can only order the disclosure of 
evidence if the claimant has undertaken all proportionate attempts at 
gathering relevant evidence from the defendant. Where a defendant fails 
to comply with an order, a national court shall presume the defendant’s 
non-compliance with a relevant duty of care, although the defendant has 
the right to rebut that presumption. 

National courts, according to the AI Liability Directive, shall 
presume!® the causal link between the fault of the defendant and the 
output produced by the AI system or the failure of the AI system to 
produce an output when: 


e the claimant has demonstrated this, or the court has presumed 
the fault of the defendant or of a person for whose behaviour the 
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defendant is responsible, consisting in the non-compliance with a 
duty of care to protect against the damage that occurred; and 


based on the circumstances of the case, it can be considered reason- 
ably likely that the fault has influenced the output produced by the 
Al system or the failure of the AI system to produce an output; and 


the claimant has demonstrated that the output produced by the AI 
system or the failure of the AI system to produce an output gave rise 
to the damage. 


The fault of the defendant will be accepted if any of the following applies: 


the AI system uses techniques involving the training of models with 
data that was not in conformity with the AI Act; 


the AI system was designed and developed in a way that does not 
meet the transparency requirements as laid down in the AI Act; 


the AI system was not designed and developed in a way that allows 
for effective oversight by natural persons as specified in the AI Act; 


the AI system was not designed and developed so as to achieve an 


appropriate level of accuracy, robustness and cybersecurity as 
detailed in the AI Act; or 


the necessary corrective actions were not immediately taken to bring 
the AI system into conformity with the obligations or to withdraw or 
recall the system as required by the AI Act. 


In case of a high-risk AI system and the defendant is a user, the fault of 
the defendant will be accepted if: 


the user did not comply with its obligations to use or monitor the AI 
system in accordance with the accompanying instructions of use or, 
where appropriate, suspend or interrupt its use; or 


the user exposed the AI system to input data under its control that is 
not relevant in view of the system’s intended purpose. 


The proposed Liability for Defective Products Directive!* would be the 
successor of the Product Liability Directive 1985.'% The Commission 
makes it clear in the explanatory memorandum that one of the three 
reasons for proposing a successor was that under the old Directive, ‘the 
burden of proof (i.e. the need, in order to obtain compensation, to prove 
the product was defective and that this caused the damage suffered) was 
challenging for injured persons in complex cases (e.g. those involving 
pharmaceuticals, smart products or Al-enabled products)’.!*° 
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The Commission underlines that the proposal makes it clear: 


e that AI systems and Al-enabled goods are ‘products’ and therefore 
fall within the scope of the Directive, meaning that compensation 
is available when defective AI causes damage, without the injured 
person having to prove the manufacturer’s fault; 


e that not only hardware manufacturers but also software providers 
and providers of digital services that affect how the product works 
(such as a navigation service in an autonomous vehicle) can be held 
liable; 

e that manufacturers can be held liable for changes they make to prod- 
ucts they have already placed on the market, including when these 
changes are triggered by software updates or machine learning; and 


e that the burden of proof in complex cases can be alleviated, which 
could include certain cases involving AI systems. 


However, there are also a number of exemptions from liability, such as: 


e inthe case of a manufacturer or importer that did not place the prod- 
uct on the market or put it into service; 


e in the case of a distributor that did not make the product available 
on the market; 


e when it is likely that the defectiveness that caused the damage did not 
exist when the product was placed on the market, put into service or, 
in respect of a distributor, made available on the market, or that this 
defectiveness came into being after that moment; or 


e inthe case of a manufacturer, that the objective state of scientific and 
technical knowledge at the time when the product was placed on the 
market, put into service or in the period in which the product was 
within the manufacturer’s control was not such that the defectiveness 
could be discovered. 


At this point in time, it is unclear how these regulations will work in prac- 
tice and would apply to synthetic technologies in particular. However, it 
is clear that the AI Liability Directive will not be able to mitigate all 
the problems with respect to applying the current standards of proce- 
dural law to these technologies, inter alia, because it is only applicable 
to matters of non-contractual liability. It could have an impact on access 
to information and the burden of proof, but leaves the other tensions 
discussed in section 5.5.2 unaddressed. For the new Product Liability 
Act, inter alia, it is clear that there is a long list of exemptions, which 
will be used and tested in court by companies in abundance. It may take 
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several years before courts will carve out the precise meaning and inter- 
pretation of the rules and exceptions in specific circumstances. 

Third and finally, internet companies play an important role in disclos- 
ing, distributing and prioritising content. This is especially relevant for 
DFs. The former regime for internet providers’ liability can be found in 
the 2000 e-Commerce Directive,” which has recently been replaced by 
the Digital Services Act (DSA).'°% The e-Commerce Directive provided 
regulations for a host of different issues concerning digital business oper- 
ations, including an exemption for liability of internet intermediaries for 
User-Generated Content. A distinction was made between three types of 
providers: access providers (Article 12), caching providers (Article 13) and 
hosting providers (Article 14). The latter were not liable for the unlawful 
activities of customers if they had no knowledge of them and if they acted 
promptly to remove the unlawful information or make access to it impos- 
sible, as soon as they are informed of illegal content on their platform. "°° 
Article 15 provided that Member States could not impose a general 
obligation on providers to monitor the information they transmitted or 
stored, or actively seek out facts or circumstances indicating unlawful 
activities. ! It is noteworthy that the e-Commerce Directive contained no 
specific provision for search engines. '*! It also had no specific provision for 
so-called ‘active internet intermediaries’, which not only passively trans- 
mit or publish users’ content, but also acquire, select, organise, rank and 
facilitate content publication. The question of whether such providers 
also fell under the Notice and Takedown (NTD) was answered ambiva- 
lently by the CJEU.!? 

After several failed attempts to replace the e-Commerce Directive, !® 
the DSA has now been adopted. Although much remains the same, 
Articles 20 and 21 deal with the possibility of internal and extrajudi- 
cial complaint handling, and Article 22 with so-called ‘trusted flaggers’, 
such as civil rights organisations that can assess online content on its 
lawfulness: 


Providers of online platforms shall take the necessary technical and organi- 
sational measures to ensure that notices submitted by trusted flaggers ... 
are given priority and are processed and decided upon without undue delay. 
The status of ‘trusted flagger’ under this Regulation shall be awarded, 
upon application by any entity, by the Digital Services Coordinator of the 
Member State in which the applicant is established, to an applicant that has 
demonstrated that it meets all of the following conditions: it has particu- 
lar expertise and competence for the purposes of detecting, identifying 
and notifying illegal content, it is independent from any provider of online 
platforms, it carries out its activities for the purposes of submitting notices 
diligently, accurately and objectively. 165 
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Article 23 sets out further obligations to counter abuse of internet 
services.'* For example, online platforms must stop providing their 
service to users who systematically abuse it and share illegal content. 
In doing so, they must take several aspects into account, including: 


e the absolute numbers of items of manifestly illegal content or mani- 
festly unfounded notices or complaints, submitted within a given 
timeframe; 


e the relative proportion thereof in relation to the total number of 
items of information provided or notices submitted within a given 
timeframe; 


e the gravity of the misuses, including the nature of illegal content, and 
of its consequences; and 


e where it is possible to identify it, the intention of the recipient of the 
service, the individual, the entity or the complainant. 


Article 16 lays down a more extensive NTD regime than that contained 
in the e-Commerce Directive, although by and large this has remained 
the same. Article 17 provides a duty for platforms to give reasons for the 
decision on a notification or complaint. Article 18 provides that plat- 
forms must inform the police and judicial authorities if suspicion of 
certain crimes emerges. 

A so-called ‘Good Samaritan’ clause,!% already contained in the US 
Communications Decency Act among others,'® is included in the DSA. 
If providers actively monitor content (eg, to detect DF child pornog- 
raphy), this cannot be used against them (eg, as an argument that by 
monitoring, they could or should also have had sight of other possible 
unlawful content).! Article 7 DSA provides: 


Providers of intermediary services shall not be deemed ineligible for the 
exemptions from liability ... solely because they, in good faith and in a dili- 
gent manner, carry out voluntary own-initiative investigations into, or take 
other measures aimed at detecting, identifying and removing, or disabling 
access to, illegal content, or take the necessary measures to comply with the 
requirements of Union law and national law in compliance with Union law, 
including the requirements set out in this Regulation. 


Given the central role that internet intermediaries play in the modern 
media landscape, some have argued that further regulation is possi- 
ble and desirable. For instance, a perennial problem is the role of 
search engines,” which have a very defining position within the digi- 
tal domain in general and for finding synthetic media in particular. 
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The NTD regime seems ill-suited for such providers. Search engines 
are active providers par excellence: they search for content on other 
pages, determine how that content is ranked and the basis on which 
this is done. Obligations could be considered that see to the disclo- 
sure of the criteria on the basis of which algorithms rank material.!7! 
Conditions could also be imposed on which criteria can be used in 
ranking content or what content should be blocked.'” This concerns 
not only search engines themselves, but also sites with search functions, 
such as Facebook, YouTube and X.!” 

It is also noteworthy that the DSA places the ultimate responsibil- 
ity for deciding on a removal request on the provider, who will have to 
judge the reasonableness of a request. This differs from the US legal 
regime, as contained for example in the Digital Millennium Copyright 
Act,”* which includes a system for ‘counter-notifications’. Under this 
system, the provider receives a request to remove material and then 
forwards it to the person who uploaded the material. If the uploader 
then claims that the material was lawfully posted, it is up to the parties 
to go to court and obtain a court order.! This system has obvious 
drawbacks, such as that taking a case to court will lead to delays.!”* 
On the other hand, it places the final choice with the court and not 
with the internet provider, which might be preferable because of their 
commercial interest, inter alia, in having sensitive and borderline-legal 
content on the platform.'”” Consideration could therefore be given to 
opening an ex parte remedy or other swift judicial procedures when 
conflicts like these arise. 

The DSA explicitly stipulates that there is no monitoring obliga- 
tion for internet intermediaries, nor is there any obligation to engage in 
ex ante content filtering.’ It is increasingly questionable whether this 
premise, which dates from the 1990s, is still valid in contemporary soci- 
ety, as it is based on the idea that internet intermediaries are neutral and 
passive. Initially, their position was likened to that of postal companies, 
which deliver mail without being responsible for their content. In princi- 
ple, a postal company does not have to check the mail it delivers in terms 
of its lawfulness. As internet intermediaries have become increasingly 
active and directive in attracting, organising, co-creating and prioritising 
content, and because the content of messages is generally available to 
them, the question is to what extent the neutrality principle should be 
applicable to the online world.!” 

Finally, it is important that Articles 33 et seq hold additional 
obligations for very large online platforms and very large search 
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180 including to conduct proper risk analysis of their services 


and take appropriate measures accordingly.'*! The responsibility lies 
primarily on these platforms to ensure that the spread of misinfor- 
mation is countered. Among the harm that the DSA aims to tackle 
is the risk that stems from the design, functioning or use, including 
through manipulation, of very large online platforms and of very 


engines, 


large online search engines with an actual or foreseeable negative 
effect on the protection of public health, minors and serious nega- 
tive consequences to a person’s physical and mental wellbeing, or on 
gender-based violence: 


Such risks may also stem from coordinated disinformation campaigns 
related to public health, or from online interface design that may stimu- 
late behavioural addictions of recipients of the service. When assessing 
such systemic risks, providers of very large online platforms and of very 
large online search engines should focus on the systems or other elements 
that may contribute to the risks, including all the algorithmic systems that 
may be relevant, in particular their recommender systems and advertising 
systems, paying attention to the related data collection and use prac- 
tices. They should also assess whether their terms and conditions and 
the enforcement thereof are appropriate, as well as their content modera- 
tion processes, technical tools and allocated resources. When assessing 
the systemic risks identified in this Regulation, those providers should 
also focus on the information which is not illegal, but contributes to the 
systemic risks identified in this Regulation. Such providers should therefore 
pay particular attention on how their services are used to disseminate or 
amplify misleading or deceptive content, including disinformation. Where 
the algorithmic amplification of information contributes to the systemic 
risks, those providers should duly reflect this in their risk assessments. 
Where risks are localised or there are linguistic differences, those provid- 
ers should also account for this in their risk assessments. Providers of very 
large online platforms and of very large online search engines should, in 
particular, assess how the design and functioning of their service, as well as 
the intentional and, oftentimes, coordinated manipulation and use of their 
services, or the systemic infringement of their terms of service, contribute 
to such risks. Such risks may arise, for example, through the inauthentic 
use of the service, such as the creation of fake accounts, the use of bots 
or deceptive use of a service, and other automated or partially automated 
behaviours, which may lead to the rapid and widespread dissemination to 
the public of information that is illegal content or incompatible with an 
online platform’s or online search engine’s terms and conditions and that 
contributes to disinformation campaigns. 18? 
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Following the European Commission’s Communication ‘Tackling 
online disinformation: a European approach’! and the 2018 Code 
of Practice on Disinformation,'+ the European Commission’s 
Communication on the European Democracy Action Plan'* and 
the European Commission Guidance on Strengthening the Code of 
Practice on Disinformation,!** in 2022, the EU has helped bring about 
a Code of Practice on Disinformation.!*’ This Code contains ambi- 
tions and commitments for the very large platforms that have signed 
it, primarily in light of monetising disinformation, such as through 
advertisements and other revenue models based on inauthentic and 
untrue content, and combating misleading and incorrect political 
advertisements. However, this Code of Practice is generally considered 
insufficient to effectively combat the rise of disinformation because 
it is too vague, has too limited a scope and is too open-ended. The 
same holds true for the proposed Regulation on the transparency and 
targeting of political advertising. !** 


5.5.5. Consumer and Competition Law 


A final way of reducing the burden on citizens (as identified in 
section 5.5.3), alongside imposing greater duties of care and responsi- 
bilities on private parties (see section 5.5.4), is through protections set 
out in consumer and anti-trust law. As the European Data Protection 
Supervisor (EDPS) has stressed, in particular in the light of new tech- 
nologies, data protection law, competition law and consumer law can 
and should be understood in light of each other, as they have partially 
overlapping goals: 


Extracting value from big data has become a significant source of power 
for the biggest players in internet markets. Not all big data is personal, but 
for many online offerings which are presented or perceived as being ‘free’, 
personal information operates as a sort of indispensable currency used to 
pay for those services. As well as benefits, therefore, these growing markets 
pose specific risks to consumer welfare and to the rights to privacy and 
data protection. EU principles and rules on data protection, competition 
and consumer protection have been designed to promote a thriving internal 
market and to protect the individual. Greater convergence in the applica- 
tion of these policies could help meet the challenges posed by the big data 


economy. 18° 
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Figure 5.5 The EDPS’ vision of the interaction between three fields of law!” 


In terms of consumer law, the EDPS points to the Directive on Unfair 
Contract Terms,”! which protects citizens acquiring advanced technolo- 
gies and data services from unintelligible and unfair contractual terms. "°? 
Importantly, the Price Indication Directive!” suggests that the price for a 
service or product should be made clear. This may be important, because 
many digital services are offered for ‘free’, while in fact consumers are 
paying with their personal data. This commercial practice may also be 
prohibited through the Consumer Rights Directive,!** which prohibits 
offering service for free when they are not. Misleading statements and 
advertisements are prohibited through the Misleading and Comparative 
Advertising Directive," while the Unfair Commercial Practices Directive 
prohibits aggressive commercial practices and harassment, coercion and 
undue influence. 16 

These provisions might put a stop to aggressive advertising and 
manipulative design choices, such as when a person is nudged into 
buying a certain product or service. Contract law!” prohibits practices 
such as abuse of circumstances or of the vulnerable position in which the 
consumer finds themself. Should a patient need care immediately and the 
HR offers a contractual service to provide such care when it is clear that 
the person is in no position to decline, the contract can be declared null 
and void. If the citizen did not fully understand the contractual terms 
they were agreeing to or had no reasonable chance of going through them, 
that will have an effect on the validity of contractual agreements as well. 
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This is an important difference with the American approach, under which, 
by and large, an informed consent model is taken. If companies include 
the most far-reaching and unfair terms in their terms and conditions and 
the consumer signs the contract, this is generally deemed to be legitimate. 
Under EU law, unfair terms are prohibited as such. Contractual terms 
that are several pages long and written in complicated legal or tech- 
nological jargon can be null and void even if they have been explicitly 
agreed upon. The consumer needs to be able to understand and refuse 
the contractual terms in order for them to be valid, and the terms cannot 
ask consent for anything unreasonable or unrelated to the contract. "8 

The European Commission has also used the instrument of compe- 
tition law regularly vis-a-vis, in particular, American-based tech 
companies, inter alia, to tackle market dominance, to block mergers and 
to prevent product tying.’ The DMA” defines gatekeepers as those 
organisations that have a significant impact on the internal market, that 
provide a core platform service which is an important gateway for busi- 
ness users to reach end-users, and that enjoy an entrenched and durable 
position in terms of its operations, or where it is foreseeable that it will 
enjoy such a position in the near future. Among many other things, these 
gatekeepers cannot: (a) process, for the purpose of providing online 
advertising services, the personal data of end-users using services of third 
parties that make use of the core platform services of the gatekeeper; 
(b) combine personal data from the relevant core platform service with 
personal data from any further core platform services or from any other 
services provided by the gatekeeper or with personal data from third- 
party services; (c) cross-use personal data from the relevant core platform 
service in other services provided separately by the gatekeeper, including 
other core platform services, and vice versa; or (d) sign in end-users to 
other services of the gatekeeper in order to combine personal data. It is 
difficult to foresee how these and the other rules contained in the DMA 
will apply to Generative AI, synthetic technologies and their potential 
consequences,””! but it is not unlikely that dominant platforms that offer 
synthetic technologies, products or services will fall under this Act. 

Legal questions that are still unanswered when approaching synthetic 
technologies through the lens of consumer and competition law include, 
but are not limited to, the following: 


e Is it misleading to advertise services as free when in fact customers are 
paying with their personal data? 

e Are the overly long and legalistic terms and conditions by American 
companies legitimate under European law or do they violate the 
requirement that citizens should be able to understand the contracts 
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they sign? To what extent will citizens ever understand the inner 
working of technologies, how their data are being processed and 
what revenue model a company is pursuing? 


e To what extent are consumers locked in when they have to accept an 
update of the terms and conditions of a service they already use? 


e To what extent are children growing up in present-day society truly 
free to decide not to engage with modern technology, given peer pres- 
sure and advantages of those that do? To what extent is parental 
consent always obtained and which duties of care should be imposed 
on platforms in this respect? 


e Do companies have a duty of care, especially with respect to children, 
to block access to services when they see that the use is harmful? 


e To what extent can the use of technologies, especially by children, be 
considered to be addictive and, if so, would that undermine the legal 
validity of consent? 


¢ To what extent are dark patterns that nudge people in the direction 
preferred by a company prohibited and to what extent is consent that 
is obtained through these patterns valid? 


e To what extent would offering synthetic technologies to people in 
need (eg, care robots that offer emergency care for additional fees) be 
considered an abuse of circumstances? 


¢ ‘To what extent is the double role of many technology companies that 
market their own products but also have a platform on which other 
vendors can offer their products allowed? 


¢ To what extent will AI programs that are implemented into existing 
services be seen as a prohibited form of product tying? 


e How should the market for Generative AI be delineated in order to 
establish market dominance? 


e To what extent are established technology companies that use their 
power and position in terms of, inter alia, access to data and custom- 
ers, to roll out their new AI products abusing their power and position? 


e To what extent are deals between companies, where the service of one 
is set as a standard tool in the product of the other, allowed in light 
of fair competition? 


e Should killer acquisitions be prohibited? If so, what should be the 
determinative factors to establish whether an acquisition is aimed at 
terminating a start-up or small scale competitor? 
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5.5.6. Conclusion 


This section has discussed rules concerning the enforcement of the regu- 
latory paradigm, the liability for violating legal principles through AI 
and the rules of procedure that apply when cases are brought before a 
court of law. It is clear that there will be many practical problems with 
ensuring adequate law enforcement in a world where synthetic technolo- 
gies have become democratised. The question is whether the primary 
focus on ex post regulation is still viable, given this and other trends. 
However, it is equally clear that many difficulties will emerge if more ex 
ante regulations are introduced. With respect to procedural law, many 
practical obstacles may arise when citizens enter into legal battles against 
tech companies or when judges assess digital evidence for authenticity. 
Equally, there will be many problems with establishing guilt or fault, 
especially when a complex web of actors (natural persons, legal persons 
and non-humans) is involved. Although the proposed AI Liability Directive, 
the proposed Liability for Defective Products Directive and the DSA try 
to provide more clarity, they also raise questions regarding their appli- 
cability and whether they are future-proof. With regard to the DSA, for 
example, it can be wondered whether imposing a general monitoring 
and filtering obligation on internet intermediaries should be imposed, 
although it is clear that this would come at the expense of the rights to 
privacy and freedom of expression of users of their platforms. Finally, 
consumer and competition law raise a number of complex questions 
concerning the validity of digital contracts, product tying and market 
dominance of companies developing and marketing Generative AI prod- 
ucts and services. 


5.6. CONCLUSION 


This chapter has discussed several legal regimes that affect synthetic tech- 
nologies and synthetic content. In relation to some aspects, the legal regime 
sets clear rules and boundaries through data protection law and the vari- 
ous legal instruments that contain rules on the production, distribution 
and use of technologies. Regulators that want to facilitate the broader use 
of synthetic technologies might consider limiting these legal doctrines. 
An example is the transparency obligation contained in the GDPR, which 
seems impossible to reconcile with the use of HRs and AR devices in public 
domains. For most other regimes, it is not entirely clear how the current 
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regulatory framework applies to synthetic technologies, because the open 
norms and principles are applied on a case-by-case basis and there is no 
jurisprudence on this matter as of yet. These include most of the rights 
contained under the human rights frameworks of the ECHR and the 
CFREU, such as the right to privacy, the freedom of expression and the 
right to property. With respect to another set of doctrines, it is uncertain 
how they will apply to synthetic technologies. How should the data quality 
principle be interpreted and how should the concept of personal data be 
demarcated in relation to synthetic media? How should psychological harm 
be interpreted with respect to AI, synthetic technologies and their applica- 
tion? Finally, there are several doctrines that might need revision because the 
rationale for adopting a doctrine no longer holds and/or because it leads to 
undesirable outcomes. This includes the household exemption, the domi- 
nant focus on ex post regulation, the presumption that evidence is correct 
unless contra-indications arise and potentially, especially in the future, the 
fact that non-human entities do not have legal standing. 


Table 5.5 Legal doctrines setting clear limitations on synthetic 
technologies and media 


Legal doctrine Limits imposed on synthetic technologies 

Transparency If personal data are gathered, data subjects must be 
informed in full. 

Purpose specification The data controller should have a specific reason 
for gathering personal data. 

Purpose limitation The data controller can subsequently only use the 
data for that specific purpose. 

Lawful ground for There must be a lawful basis for processing 

processing personal data, such as consent or a legal basis. 

Security and Third-party access must be prevented; data need to 

confidentiality be safe 

Interoperability At the request of the data subject, data must be trans- 
ported to a new provider so that data are re-usable. 

Export prohibition When technologies or data can be abused for 
human rights violations, export prohibitions may 
be applicable. 

Accuracy and AI needs to produce accurate and reproducible 

reproducible results. 

Accounting and Providers and distributers must put in place 

oversight adequate mechanisms for documentation, evalua- 


tion and oversight. 
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Table 5.6 Legal doctrines with effects on synthetic technologies and media that 
will be determined on a case-by-case basis 


Legal doctrine 


Case-by-case basis analysis 


Privacy as personality 
right 


Article 8 ECHR provides a broad personality right 
to flourish as a human being; to what extent this 
applies to synthetic technologies will be assessed 
on a case-by-case basis. 


Chilling effect 


Article 8 ECHR offers protection against a chill- 
ing effect; to what extent such an effect exists and 
meets the de minimis rule will be assessed on a 
case-by-case basis. 


Reasonable expectation 
of privacy 


Article 8 ECHR may conflict with the freedom 
of expression; whether citizens have a reasonable 
expectation of privacy vis-a-vis online material 
and public conduct will be determined on a case- 
by-case basis. 


Legitimate expectation 
of privacy 


Article 10 ECHR may conflict with the right to 
privacy when a publication about a public figure 
contains intimate details; whether the public figure 
has a legitimate expectation of privacy will need 
to be determined on a case-by-case basis. 


Property and business 
rights 


Several parties may hold various property rights to 
synthetic technologies and media; how these play 
out against each other will be determined on a 
case-by-case basis. 


Procedural fairness 


Citizens need to be informed and involved in deci- 
sions that affect them; the extent to which these 
principles apply and have been respected will be 
assessed on a case-by-case basis. 


Consumer and 
competition law 


How the various standards developed under 
consumer and competition law apply to 
Generative AI will be evaluated on a case-by- 
case basis. 
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Table 5.7 Legal doctrines whose interpretation vis-a-vis synthetic technologies 


and media is unclear 


Legal doctrine 


Uncertainty as to application 


Personal data 


The GDPR places a binary distinction between 
non-personal data and personal data, while the 
way in which synthetic technologies and media 
relate to a specific person depends on gradual 
factors and sliding scales. This tension applies 
more in general to the choice of the EU regulator 
to work with several distinct and clearly defined 
data categories, which it has attributed different 
levels of protection. 


Data quality 


The data quality principle seems difficult to apply 
to synthetic media. 


DF transparency 
obligation 


Physical or psychological 
harm and detriment 


Mental and psychologi- 
cal integrity 


How the DF transparency requirement in the AI 
Act should be interpreted and relates to the trans- 
parency requirement in the GDPR is not yet clear. 


How various terms in the AI Act and other EU 
technology regulations, such as harm and detri- 
ment, should be interpreted considering synthetic 
technologies and media will need to be clarified in 
the future. 


There are various doctrines in the jurisprudence 
of the ECtHR, the DSA, the AI Act, the CFREU 
and other legal instruments that aim at safeguard- 
ing mental and psychological integrity. There is a 
lack of clarity as to their application and effect on 
synthetic technologies. 


Liability 


Exactly how the current rules for fault and 
liability should be interpreted considering the 
responsibility for non-human actors in a complex 
web of parties involved is unclear. 


Authentic evidence 


Exactly how the prevailing rules on evidence and 
their authenticity should be interpreted consider- 
ing synthetic media and AI detection programs is 
uncertain at this point. 
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Table 5.8 Legal doctrines that may need to be reconsidered in light of synthetic 


technologies and media 


Legal doctrine 


Potentially in need of revision 


Household exemption 


It is questionable whether the household exemp- 
tion should be maintained in its current form, 
as the boundaries between the private and the 
public sphere are less and less absolute. 


Automated 
decision-making 


It is clear that limits on automated decision 
making are more relevant than ever, while the 
provision in the GDPR is framed using such 
narrow wording that it plays almost no signifi- 
cant role in practice. 


Human rights for non- 
human entities 


A question is whether non-human actors should 
be attributed rights and legal standing. 


Discrimination and bias 


The current doctrine on discrimination seems 
unsuited to counter the many problems of bias, 
Matthew effects and systemic discrimination 
connected to the use of AI on many accounts. 


Disinformation 


It is a matter of debate whether telling untruths 
and disseminating disinformation should be 
covered by the freedom of expression or should 
be sanctioned, even when no direct harm is 
done. 


Democracy 


There are arguments to be made in favour of 
extending legal protection of civil servants and 
politicians against synthetic technologies, and 
against those technologies (in particular DFs) 
being used during election campaigns. 


Importing data and 
technology 


There are relatively few rules on importing data 
and technology into Europe, although this might 
be necessary in light of synthetic technologies. 


Active hosting providers/ 
search engines 


There are pleas to revise the rules exempting 
active internet providers from liability, as these 
rules were designed for passive intermediaries. 


Ex post regulation 


Although the EU has proposed several regimes 
that introduce ex ante rules, by and large, the 
current legal regime consists of ex post regula- 
tion. It can be questioned whether this is viable, 
given the pressure this puts on law enforcement 
organisations and on citizens to effectuate their 
rights. 
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Uneasy Questions and 
Imperfect Solutions 


Question Zero 


Due to the invasiveness of some Al-applications or uses, there might be situ- 
ations in which our current framework of human rights, democracy and the 
rule of law fails to adequately or timely protect us and where we might need 
to pause for reflection and find the appropriate answer to what one could 
consider “question zero”: Do we want to allow this particular Al system or 
use and if so, under what conditions? Answering this question should force 
us to look at the Al-system or use from all perspectives, which could result 
in several ‘solutions’: 


¢ A particular Al-system or use is put under a moratorium, (temporarily or 
indefinitely) banned or put under restrictions (“Red Lines”) 

e New Human Rights are introduced as safeguards against the ‘new’ 
adverse impact of AI 

¢ Existing Human Rights are adapted to allow for responsible development 
and use of Al 

¢ A particular Al-system or use is made subject to a specific democratic 
oversight mechanism 

e Private owners of powerful Al-systems are obliged to align their AI devel- 
opment and governance structures with the interests of those affected by 
the system and society at large, which could include measures to involve 
relevant parties (such as workers, consumers, clients, citizens, policy 
makers) 


First and foremost, ‘Al impact’ is to be considered both at individual and 
at societal/collective level whereas Al can impact both the individual as 
well as larger parts of our collective society. Secondly, context, purpose, 
severity, scale and likelihood of the impact is important to determine the 
appropriate and proportionate action. For Al applications that generate 
unacceptable risks or pose threats of harm or systemic failure that are 
substantial, a precautionary and principle-based regulatory approach 
should be adopted. For other Al applications a risk-based approach could 
be more appropriate. 


Council of Europe Study 
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6.1. INTRODUCTION 


HIS FINAL CHAPTER provides an overview of the lessons that have 

been drawn thus far, and provides a glimpse into what lies ahead. 

It first summarises the main findings of the five previous chap- 
ters (section 6.2) and then maps some of the proposals that experts have 
suggested for dealing with the moral, legal and societal questions raised 
by synthetic technology (section 6.3). It continues to detail the myriad 
complex dilemmas and intricacies that legislators need to account for, 
sometimes necessitating a choice between important yet conflicting soci- 
etal interests (section 6.4). A model is then offered for approaching regu- 
latory decisions when dealing with Synthetic Reality (SR), setting out 
what questions need to be dealt with and in which order when deciding 
over legal frameworks for Humanoid Robots (HRs), Deepfakes (DFs), 
Augmented Reality (AR) and Virtual Reality (VR) (section 6.5). Finally, 
a short conclusion is provided (section 6.6). 


6.2. MAIN FINDINGS 


Chapter 1 of this book shed light on the concept of SR and the four tech- 
nologies that are central to this book. It exemplified the broad societal 
implications these technologies may have by analysing the effects of a 
small, hypothetical incident involving sensational news which may have 
been created through DF-technology. It also gave an overview of this 
book’s content, as well as listing its many limitations. It pointed out, inter 
alia, that the terms ‘Humanoid Robots’, ‘Deepfakes’, ‘Augmented Reality’ 
and ‘Virtual Reality’ do not lend themselves well to exact definitions, that 
there is overlap and that they are not always clearly distinguishable. It 
explained that, by the time this book is published, a number of concrete 
examples offered in it may be outdated. The point was emphasised that 
because these technologies are still in their relative infancy, most sources 
currently available that cover the effects of synthetic technologies come 
from companies that develop them, from computer scientists who work 
on perfecting and operationalising AI, and from tech-savvy journalists and 
academics. Consequently, the sources may be biased towards benevolent 
use cases and positive expected impact and efficacy. Chapter 1 also stressed 
that there is a real chance that these technologies, even Al as such, will not 
take off or will not become the success that they currently projected to 
be, just as the previous AI revolutions have died quiet deaths previously. 
Finally, it was explained that this book is written for a broad audience, 
but caters in particular to lawyers, legal scholars, regulators, and law and 
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technology students, meaning that the technologies and their effects in this 
book are described through that lens. 

Chapter 2 provided an overview of how synthetic technologies are used 
in practice. Although these use cases are typically divided into positive and 
negative ones, this classification is obviously not uncontroversial. The lines 
drawn are dependent on, inter alia, ethical, societal and political beliefs. Is 
having Napoleon teach history classes to children good in terms of engage- 
ment or bad because it accustoms them to living in a post-truth society? Is 
a politician speaking in the dialect or language of every minority in their 
constituency good in terms of inclusion or a modern form of voter deception? 
Is being able to speak to a deceased partner good for the mental health or 
bad because it disrupts the normal path of grief processing? Is it good if the 
police use child avatars and DF child pornography to infiltrate paedophile 
networks or bad because it involves producing child pornography? 

The use cases discussed in Chapter 2 were roughly categorised into 
five domains. 


e Inthe medical sector, synthetic technologies are used, inter alia, for 
educational purposes, such as watching a dissection in VR. HRs are 
being deployed for non-patient-related tasks, such as cleaning and 
disinfecting, as well as patient contact in elderly care. DFs are used 
for grief loss therapy. AR can assist surgeons in precision tasks. 


¢ In industry and commerce, among many other things, HRs are used 
for replacing human jobs. AR is being used for Al-assisted manufac- 
turing. Both AR and VR enable new forms of tourism and, finally, 
DFs allow for virtual fitting rooms. 


e For educational purposes, DFs can enable historical figures to teach 
history lessons to schoolchildren. AR is used for bringing school- 
books to life. Home schooling using VR was popular, especially 
during the COVID-19 pandemic. 


¢ Inthe entertainment industry, VR is used mainly for gaming. DFs are 
popular for satire. AR can be used for having an avatar join a party. 
All technologies are used for erotic purposes, such as through teledil- 
donics, DF porn and humanoid sex robots. 


e Synthetic technologies are used for security-related purposes, such as 
having police officers patrolling the streets wear AR glasses allowing for 
facial and emotion recognition, having HR soldiers engage in combat or 
training military personnel in VR. DFs are increasingly popular as part 
of military strategies to spread misinformation about the opponent. 


Many of these applications will have positive effects. Virtual travelling 
can reduce carbon emissions, while HRs can perform some tasks more 
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efficiently and replace humans in arduous and degrading jobs. SR can 
open up new ways for people to explore their identities and reach out 
to others. However, it is equally clear that synthetic technologies can be 
used for a wide range of detrimental purposes and can have unwanted 
effects. Synthetic technologies are used to produce fake child pornog- 
raphy and non-consensual pornographic material. Synthetic gaming 
applications may worsen addictions, especially for children, and may 
exacerbate feelings of loneliness and lead to bodily neglect. Given the 
immersive nature of SR, the vulnerabilities innate in this kind of tech- 
nology create unprecedented opportunities for third parties (eg, hackers 
or malign family members) to interfere with people’s lives, such as when 
HRs are used in medical settings or for care purposes, or when patients 
are treated for physical or psychological trauma in VR. 

Chapter 3 provided a sketch of the technical design and architec- 
ture of synthetic technologies. It explained that the ideal of developing 
HRs is to have them become indistinguishable from human intelli- 
gence and appearance. Most contemporary HRs excel in only one or 
two aspects, such as in their physical appearance, their social skills 
or their ability to perform one specific task. However, holistic robots 
are being developed and Artificial General Intelligence? is loom- 
ing on the horizon. DFs aim to be similar to authentic material on 
a representational level. When deployed for satire, DFs are often so 
remarkable and odd that the consumer knows they are fake, but other- 
wise they are generally undistinguishable from authentic material. 
Over time, it may become almost impossible to tell what material is 
real and what is fake, as even AI detection tools are only partially 
effective in this respect. With AR, consumers are typically aware of 
the synthetic nature of the reality that is presented to them, although 
with subconscious and micro-augmentation, this may not necessarily 
be so. Finally, in VR, physical reality is replaced in full. The virtual 
world may resemble Physical Reality to varying degrees, which also 
includes the user’s avatar. VR allows the user to shape their avatar 
according to their idealised body image. Some people already identify 
more with their virtual avatars than with their physical bodies, prefer- 
ring the virtual world over the analogue one. 

SR, although potentially tapping into a whole range of stimuli, is 
currently mainly focused on visual cues. Audio is also of great impor- 
tance in HRs and DFs, and AR and VR too in some cases. Olfactory and 
haptic input and output systems are currently underdeveloped, although 
with HRs, haptic cues are becoming increasingly important, especially 
through what is known as ‘soft robotics’, and AI scents generators are 
in the making. Gustatory sensory input and output systems are largely 
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absent for any of the four technologies, although they are conceivable 
for HR and VR. DFs and AR are generally one-directional, although 
there are exceptions, while VR and HRs typically depend to a significant 
degree on their interactional faculties. 


Resembling reality 


ost advanced DFs; 


dvanced chatbots; 
highly advanced VR 
resembling real 
world; ideal stage 
for HRs 


ibconscious impact; 
Rs performing one 
task 


One-directional Interactional 


VR dream world; 


applications 


Virtual tour guide) 


Clearly inauthentic 


Figure 6.1 Categorisation of the four techniques on two axes (interactionality 
and reality) 


Chapter 4 dealt with the societal effects of SR, focusing primarily on the 
potential long-term effects and the broader social and moral implica- 
tions of AI. Because these effects will only come to light in the future, 
if at all, this chapter was more suppositive than the previous two. Many 
of the potential dangers that were discussed can be seen as two sides of 
the same coin, which may materialise simultaneously. For example, one 
risk of SR is that because of synthetic media, people’s reality and their 
perception of it may change radically and suddenly. For example, the VR 
world where they spend much of their life may be radically redesigned, 
an avatar might be stolen or an HR with which a person is intimate may 
reprogram itself. However, there is also a risk that people withdraw into 
static dream worlds, seeking confirmations of their established world- 
views. This could be done for a range of reasons: to keep talking to a 
deceased spouse; fleeing to VR to escape an unpleasant physical reality; 
or only consuming media that confirms or conforms to fixed beliefs. To 
give another example, on the one hand, there is a danger that robots and 
avatars will become able to replace humans at work, and, on the other 
hand, an equal danger that they will never become truly equal, leading 
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to recognition processes between two unequal partners (eg, men who 
build a life with their sex robot or children partially raised by Al-driven 
entities). Yet another example is that AI will have a consolidating effect 
on reality — for example, because it reproduces consensus knowledge 
rather than contrarian views — but also that AI will fabulate and falsely 
present its fantasies as facts. To give a final example, there is a potential 
Matthew effect: through the unequal distribution of synthetic technolo- 
gies, the rich may get richer while the poor get poorer. However, equally, 
it has been pointed out that when AI relieves a person of many of the 
ordinary, complex and tedious daily tasks, they may be hampered in 
terms of developing a full-fledged personality because this requires over- 
coming setbacks and hardship. 

With regard to other points, the dangers seem to point clearly in 
one direction. Synthetic media can accelerate the post-truth society 
and fuel polarisation, undermining democratic elections and decision 
making. It can also have a significant impact on court proceedings 
and media reporting, as it may become increasingly unclear what 
content or evidence is authentic and what is synthetic. The over- 
flow of synthetic media may also lead to ‘reality fatigue’: a genuine 
disinterest in the question of whether something is real or not. Since 
synthetic technologies are mainly controlled by a small number of 
private sector organisations, there is a danger that they will gain an 
unprecedented level of control over what is perceived to be real. People 
will also become more dependent on technology. Although technol- 
ogy may enhance autonomy (eg, helping blind people to navigate the 
streets), this dependency may ultimately undermine human autonomy, 
as these technologies are generally marketed by firms with commer- 
cial interests. Individual autonomy may also be in jeopardy because 
through synthetic technologies, third parties can gain influence over 
what people are able to do, and AI may be programmed to prohibit 
certain behaviours and speech. 

Finally, Chapter 5 discussed several legal regimes that affect synthetic 
technologies and synthetic media. This chapter focused on the existing 
European regime as well as concrete proposals that have been made for 
regulating AI and aspects directly or indirectly relevant for synthetic 
technologies. The Council of Europe’s legal acquis is primarily relevant 
in terms of human rights, such as the right to privacy, freedom of expres- 
sion, non-discrimination and intellectual property. The EU has adopted 
and proposed laws on data protection (the General Data Protection 
Regulation), AI (the AI Act), liability (the AI Liability Directive, the Product 
Liability Directive and the Digital Services Act) and a wide variety of laws 
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on the production, import and export of data and technology (eg, the 
Dual Use Regulation, the Machine Regulation and the Data Governance 
Act). Four categories of legal principles were identified. 

First, legal doctrines that set clear limits on synthetic technolo- 
gies. The GDPR, for example, requires that everyone whose data are 
processed should be informed in full and be given the right to object. For 
many synthetic applications, it is not clear how it would be possible in 
practice to adhere to these requirements. The GDPR also requires that 
all collected personal data serve a specific, predetermined purpose and 
that only the data that are strictly required in light of that purpose may 
be gathered, which may raise complicated questions, such as when AR 
devices and HRs operate in public and semi-public spaces and gather 
data by default. Also, the GDPR provides that data may subsequently 
only be used for their preselected purpose, which may significantly limit 
the reuse of data for generating DFs and for projecting personal infor- 
mation onto AR devices. Some of these limits may place insurmountable 
hurdles in the way of several concrete applications of synthetic technolo- 
gies. To allow for these applications, the regulator would need to change 
the GDPR, the AI Act and the Framework for Trustworthy AI, among 
other instruments. 

Second, there are doctrines, especially within the human rights 
framework, which are termed in such general ways that what they mean 
for synthetic technologies and concrete applications is not immediately 
clear. These principles include, inter alia, the ECtHR’s emphasis on 
the protection of personality rights, safeguards against chilling effects, 
privacy protection (both at work and in the public sphere), guarantees 
for property rights and the various procedural safeguards that need to 
be respected when decisions are adopted that have an impact on human 
rights. One complicating factor is that the ECtHR usually only provides 
a legal interpretation on a case-by-case basis, meaning that the applica- 
tion of the various principles varies based on the case, application and 
setting. Although this allows for flexibility, it comes at the cost of legal 
certainty and foreseeability. Both tech organisations and citizens, as well 
as supervisory authorities, would benefit from more regulatory clar- 
ity, which the legislator could provide through guidelines and sectoral 
frameworks. 

A third category is where it is unclear how current legal doctrines 
could, would or should be interpreted in relation to synthetic technolo- 
gies, not so much because they provide general norms, but because they 
have been written with a different reality in mind. For example, what 
does the data quality principle mean in light of synthetic content? 
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How will the rules on liability be applied to the complex web of inter- 
national actors that each have their own role in the value chain? How 
should the rules on evidence, authenticity and burden of proof be inter- 
preted in a synthetic society? There are many more questions like these. 
Consequently, further regulations or bylaws may need to be adopted to 
shed light on what these principles entail for synthetic technologies, or 
they may need to be redesigned. 

The fourth and final category consists of legal doctrines that have a 
protective aim, but seem unable to deliver on that aim in a synthetic soci- 
ety or come with significant costs. For example, the household exemption 
was proposed in the early 1990s when exempting data processing in the 
private sphere from the regulatory scope of data protection law seemed 
intuitive and necessary to protect the privacy of citizens. Now, however, 
citizens can produce highly realistic DF pornography of others from their 
home computer and distribute it to a global audience at the click on the 
mouse. Also, the standards developed under anti-discrimination law are 
ill-suited to the AI context. More generally, the current regulatory frame- 
work is almost exclusively focused on ex post regulation. This means 
that rights can only be invoked and supervisory authorities can only act 
when technology has been distributed and used and has caused concrete, 
demonstrable and individualisable harm. Owing to the high impact of 
synthetic technologies, not only on individual interests but also on general 
interests and society at large, it may be advisable to prevent harm and set 
limits on the production, distribution and use of technologies beforehand. 
This approach seems to be foreshadowed in part by the AI Act. However, 
the AI Act is largely procedural in nature and only sets substantial prohi- 
bitions on the use of AI in a very limited number of cases. 


6.3. LITERATURE OVERVIEW 


It almost seems as if, at this point, all of the world’s sharpest minds, 
smartest regulators and leading experts are working on this one topic: 
understanding Generative AI and its implications for the synthetic society 
from legal, ethical, technological, sociological, psychological and other 
perspectives. This book is heavily indebted to their insights, critiques and 
solutions. This section discusses several of the leading experts who have 
contributed to this rich debate in order to provide academics with a start- 
ing point for delving further into selected topics and to present regulators 
with building blocks for new regulatory approaches. The regulatory 
questions Generative AI raises are largely the same within and beyond 
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Europe, which is why regulatory critiques and proposals from non-Euro- 
pean experts will also be highlighted. Because covering all of the relevant 
literature would require a book of its own, this section can only provide 
an incomplete and selective overview that does not do justice to the many 
groundbreaking understandings, theories and suggestions available in 
academic literature and beyond. 

A first important topic that has attracted considerable attention 
concerns explainable AI (XAI).° Pasquale signalled early on that those 
who deploy AI are often unable to understand or explain its processing 
mechanisms and outcomes.’ This problem, as highlighted by Arrieta 
et alf and Samek et al,° becomes even more acute with self-learning 
Al, as systems can reprogram themselves in ways that their design- 
ers cannot foresee. Colaner has shown that one of the reasons why 
the field of XAI is so complex is that what needs to be explained by 
who to whom is a matter of debate and circumstantial.” Gonzalez 
Fuster echoes this point and suggests that, essentially, any explanation 
requires some form of translation. This means that explanations need 
to be tailored to its specific audience and setting. Others underline that 
an explanation of Al-produced results should be provided by a human 
rather than the AT itself.? Phillips et al provide four principles to which 
XAI should adhere: 


e Explanation: a system must be able to provide evidence or reasons for 
its processes and outcomes. 


e Meaningful: these explanations should be understandable to the 
intended audience; 


e Accuracy: these explanations should be accurate and all-encompassing; 


e Limits: a system may only operate under conditions for which it was 
designed and when there is sufficient confidence in its output. 1° 


Many experts also emphasise the need for contestability, as there is a 
tendency to quickly adopt and follow AI outcomes because of a false 
belief in the exactitude of AI systems, because of concerns for efficiency 
or because people do not want to take moral or legal responsibility 
for decision making.!! Clearly, contestability is of the utmost impor- 
tance in the medical domain, as Ploug and Holm have shown,” but as 
Almada argues, contestability may need to be a feature of all AI systems 
and be a way to get humans back into the loop." Hildebrandt, to give 
a final example, builds on the work of Mouffe! and Crawford, and 
has called for what she terms an ‘agonistic form of machine learning’, 
going beyond agnostic forms of machine learning. She points out that 
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the new manipulability that comes with computational inferencing 
requires rethinking democratic theory as well as the practice of assessing 
technologies. 16 

A related problem is the consolidation and reaffirmation of person- 
alities, power and truth through AI. Even back in 2011, Pariser warned 
about society fracturing into filter bubbles and echo chambers,” a danger 
that many, such as Schmidt, have since pointed out is all the more evident 
in a synthetic society.!® In order to move away from being captured in 
profiles, it is possible to embed serendipitous outcomes in systems by 
design, as Reviglio argues.!? Others like Wang et al have essentially 
proposed solving this problem by granting users more control,?? which 
may be more effective, as the extent to which serendipity can be auto- 
mated is a matter of debate.*! 

Explainability and contestability are essential for multiple values, 
such as allowing supervisory authorities to exert control and to ensure 
output accuracy; they are also preconditions for human autonomy. 
Autonomous AI systems and human autonomy are often contrasted. 
However, as Prunkl shows, in reality, this picture is more complex.” 
Susser, Nissenbaum and Roessler discussed the manipulative effects 
of AI at length.” Thaler and Sunstein have written a seminal work 
on nudging,” which Yeung has reworked for the Big Data era as 
hypernudging.” Laitinen and Sahlgren suggest that considerations about 
autonomy can be located in different constitutive dimensions, ranging 
from the initial adoption of a given technology to broad cultural and 
societal effects resulting from large-scale use.” Van de Poel has argued 
that principles ensuring human autonomy should be embedded in the 
technological design of systems.” This is a thought that has given rise 
to the term ‘human-centred AI’, which has been interpreted in a wide 
variety of ways, but often includes respect for human autonomy, as 
Shneiderman and other authors have shown.” Others, such as Verbeek, 
have argued that technology can enhance human autonomy,” sometimes 
captured by the term ‘extended autonomy’. 

Many experts have signalled the need for changes to non-discrimi- 
nation law in order for it to retain relevance in light of AI.” Zuiderveen 
Borgesius has called for a more sector-specific approach to Al-induced 
discrimination and bias.?! Wachter has suggested broadening the cate- 
gories of data that are deemed discriminatory, including the protection 
of non-traditional groups.’ Custers has focused on group profiling. 
Others, such as Mantelero, starting from this point, have called for 
giving more prominence to group protection, instead of the current 
limited focus on individual harm.*+ Xenidis and Gerards have pointed, 
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among many other things, to the dynamic nature in which AI systems 
categorise people, with constantly changing and evolving groups.” Veale 
and Binns, to give a final example, have mapped ways in which to make 
AI fairer without the use of sensitive data.** 

It is important that, on all of these points, AI may create not only 
dangers but also opportunities. It is not as if current, human decisions 
are without bias. On the contrary, as Frank et al and many others have 
shown, humans are generally biased on multiple accounts.*” The same 
applies to being able to provide explanations, contestability and trans- 
parency. Clearly, there is a danger that AI systems are unexplainable, 
non-contestable and opaque. However, the current, human-centred status 
quo is far from ideal as well. Many decisions made by humans are based 
on subconscious processes; post hoc rationalisations provided for those 
decisions often do not explain the real reasons behind them. AI systems, 
when built in the right way, may consequently help to remove the bias 
from decision making, to make decisions more objective and rational, 
and to provide more accurate information to various audiences.°** The 
same is true of serendipity: humans often seek sources, situations, jobs 
and social contacts that fit their pre-established views and beliefs. AI 
may, if so programmed, increase diversity and consequently potentially 
decrease isolation and polarisation. 

Trust is a much-discussed topic in the AI community too. Keymolen 
has shown how essential trust is for human interactions, but also how 
tech companies are deliberately designing technologies so that they seem 
trustworthy, when in fact they are not.? Waldman has underlined the 
deep relation between trust and privacy. There is a general consen- 
sus regarding the discrepancy between the trust consumers have in tech 
companies and the actual level of responsibility they take. Two ways have 
been suggested to close this gap. One is to make technologies themselves 
more trustworthy. This has led to a body of literature on trustworthy 
AI.*! Floridi and many others have made suggestions on this point, 
stressing the broader need to have ethics inform law and policy making.” 
The other is to impose more duties on tech companies. Balkin has termed 
these ‘fiduciary duties’,*? a concept that has been expanded upon by 
Richards and Hartzog,** Dobkin** and Jones, Rubel and LeClere,** and 
others. 

Besides these debates about AI and how to regulate it, there are also 
notable discussions about the impact of AI on specific legal domains. 
For example, a debate that has been playing out for two decades, as can 
be found in Bygrave’s early works*’ concerns the definition and scope of 
‘personal data’ as a concept. This debate has intensified with the rise of 
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new technologies. In a famous article from 2009, Ohm showed that data 
can either be entirely anonymous or useful, but never both, meaning that 
if organisations process data they find valuable, they should be deemed 
to fall under the scope of the data protection framework.*® This is all 
the more true in the current AI environment, in which it is increasingly 
easy to link non-personal, anonymised and aggregated data back to an 
individual or selected group.” Finck and Pallas have found that as of 
yet, a harmonised approach to the definition of anonymisation is lack- 
ing.® While Graef, Gellert and Husovec*! argue that the lines between 
personal and non-personal data are increasingly difficult to draw, and 
others suggest that it might be desirable to bring the processing of non- 
personal data partially under the scope of data protection law,” Purtova 
argues that the concept of personal data has been stretched too far and 
that it should be redefined based on different factors. 

Regarding the legal and ethical status of Al-driven entities too, a big 
debate among scholars, policy makers and experts is unfolding.*4 Darling 
has written a book on how to understand robots, drawing parallels and 
differences between animals, humans and robots.** Gellers has explored 
questions about the attribution of rights to robots,” as has Gunkel.’ 
Rodogno discusses what moral and rational capacities robots must have 
in order to be considered moral agents and patients,’ a path that is also 
trodden by Shevlin.’ Wallach and Allen describe ways to make machines 
moral agents, an approach also suggested by Sharkey.°! 

On the question of mental integrity, a new field is emerging, which 
is sometimes captured by the terms ‘mental privacy’ and ‘cognitive 
privacy’.® Ienca and Malgieri, for example, focus on technologies that 
can map neural interfaces and interpret mental processes, calling for 
a ‘Mental Data Protection Impact Assessment’, a specific data protec- 
tion impact assessment designed to better assess and mitigate the risks 
to fundamental rights and freedoms associated with the processing of 
mental data. Others, such as Oravec,® have focused on modern forms 
of lie detection. Emotion detection would, if the dreams of the devel- 
opers are realised, allow for the detection of emotions before a person 
themselves is aware of them. This is generally deemed to be a particularly 
invasive tool.®° This has led Roemmich, Schaub and Andalibi to call for 
the protection of emotional privacy” and Bard for a legal framework on 
this issue.°* 

The effects of synthetic technologies and AI on intellectual property 
are approached in multiple ways.® Kop discusses, among many other 
things, the matter of legal personhood of Al-driven entities.”? Hilty, 
Hoffmann and Scheuerer suggest that AI may even have an effect on 
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the philosophical underpinnings of the intellectual property regime.”! 
There is an intense academic debate surrounding the ownership of 
content produced by AI systems, as well as patent and property rights 
on AI systems; a small overview of this is provided by Omorov.” Xue et 
al discuss ways to detect and correct intellectual property infringements 
by building high-performance deep neural networks specifically trained 
for this purpose.” Post-mortem privacy is a matter that exists some- 
where between the right to privacy and the right to intellectual property; 
the question of what happens to one’s data, image and likeness after 
one’s death is a topic on which Harbinja in particular has been prolific.”4 
Finally, Chesney and Citron indicate that services might arise in light of 
SR that allow for life logging, to ensure that an exact copy of someone’s 
life is available, among many other topics.” 

Many have written about the impact of AI on the right to freedom 
of expression.”© One question that is particularly acute, as DeVries has 
pointed out,” is the extent to which chatbots fall under the protective 
scope of the freedom of expression. Massaro and Norton look at issues 
regarding speech rights in relation to virtual assistants.” Others have 
focused on how AI can be instrumentalised to enforce current legal and 
professional standards. Llansó, for example, discusses how AI can be 
used for content moderation;”? Heller signals the possibilities of using AI 
for preventing extremist content from spreading online? and Helberger 
et al stress, inter alia, that internet platforms could use AI to create more 
pluriform and diverse media landscape.*! Marsden and Meyer, and 
Bontridder and Poullet have touched on the extent to which spreading 
fake news falls under the freedom of expression.** Nuñez analyses to 
what extent platforms should be held liable for not removing inaccurate 
content. 

There is a large amount of literature concerning the design of Generative 
AI and synthetic technologies, especially from the technological side. 
Urquhart discusses the privacy and security vulnerabilities of loT-operated 
devices, as well as the lack of interoperability between and common stand- 
ards for these devices.*+ The need for more and better interoperability is a 
common theme in the literature. A large international team of researchers, 
led by Rehm, has signalled the need for greater interoperability of both AI 
and language technology, indicating five levels of interoperability, the final 
one being a world in which there is a fully realised and automated AI devel- 
opment workflow, enabling the automated development of new AI tools 
by providing fully interoperable data and tool exchange pipelines.®> Lehne 
et al have explained how essential interoperability is for digital medicine, 
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especially in light of using, interpreting and selecting data, and avoiding 
mistakes in terms of results and diagnosis.*® 

Maurushat and Hamper echo the technical vulnerabilities raised 
by Urquhart and signal the considerable marketplace for those vulner- 
abilities (eg, zero-day exploits).8’ Osena et al point out that there are 
not only challenges in terms of privacy and security, but that AI may 
also present opportunities for guaranteeing higher levels of confidenti- 
ality, privacy and system security.*® This is in line with the research by 
Hukkelas, Mester and Lindseth, who extensively discuss the possibilities 
of what they call ‘DeepPrivacy’: the protection of privacy and anonymity 
through DF technology.” 

The topic of import and export controls on data and technology 
is widely discussed in the literature,” especially after regulators took 
steps in that direction, such as the deal between Japan, the Netherlands 
and the US to set limits on the export of semiconductor machines to 
China,?! and the move away from Huawei-powered Wi-Fi networks in 
Europe.” Flyn has discussed the need for more export controls on AI in 
the US,” and from the European perspective, it is interesting to see that 
many experts, such as Carrozza, Marsh and Reichberg, and Barker and 
Hagebdlling, discuss the need for additional import and export controls 
on national levels, potentially signalling that the supranational legisla- 
tion on this point is insufficient.” 

One related topic is the discussion concerning the need to ban 
technologies and applications. Generally, European academics seem 
more welcoming of the idea of imposing limits on technologies than 
Americans and experts from other parts of the world. However, there 
has been a famous call for a moratorium on the development of LLMs 
from a large group of primarily American professionals.’ This call has 
attracted a broad response from many, including Ienca.?° The implica- 
tions of moratoria of facial and emotion recognition in public spaces 
have been discussed, by Gilbert,” while Castelvecchi has evaluated a 
potential total ban on facial recognition?’ and others have suggested 
prohibiting voice cloning technology. The EU AI Act’s limitations on the 
use of AI by law enforcement authorities, among other topics, has been 
discussed by, among others, Kaminski” and Ebers et al. 

Finally, there is a large amount of literature on the use and regulation 
of Al in specific fields, sectors and domains. 

In the medical domain, Terry has provided a good overview of the 
requirements for the use of AI in medicine, as well as the need for addi- 
tional controls.!°! Pesapane et al have focused on radiology and have 
pointed to the different legal regimes in Europe and the US which may 
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hamper the interoperability and scalability of machines. Of all the 
uses of AI in the medical profession, predictive analytics is perhaps in 
the most need of strict regulation, as prediction always entails over- and 
under-inclusion, which can have significant effects on patients, as Parikh, 
Obermeyer and Navathe have pointed out.!® Others, such as Beckers, 
Kwade and Zanca, have further explored what EU medical device regula- 
tion means in practice.!* O’Sullivan et al have studied the regulation of 
autonomous robotic surgery." and Venkatesh, Raza and Kvedar have 
considered the opportunities, dangers and potential need for additional 
regulation in light of the creation of virtual health representations (digital 
twins) of patients (physical twins) that are generated using multimodal 
patient data, population data and real-time updates on patient and envi- 
ronmental variables. 106 

In the security sector in particular, the opportunities of AI!” as well 
as the need for further regulation are mentioned by many. Although 
there are a number of experts who discuss the use of military AI in 
China,!©? Russia! and the US," it is clear that Europe is lagging behind, 
as highlighted among others by Soare: there is no unified military strat- 
egy in Europe.!' Ulnicane refers to another potential problem: the EU 
wants to be a global military AI leader, but, at the same time, wants to 
do so in a normative and ethical fashion. These are two goals that may 
not be entirely compatible.''? There is remarkably little about the use of 
‘robonauts’ and the regulation surrounding the use of AI in space. This 
may be because these applications are relatively new, perhaps because 
space is left largely unregulated, and arguably because space activities 
are generally classified, although authors like Soroka and Kurkova,!™“ 
and Martin and Freeland!" have provided good overviews of the various 
complicated questions in this field. 

A final sector that is widely discussed is the use of synthetic tech- 
nologies for sexual applications, such as DF pornography, AR devices 
that allow a person to see anyone they encounter naked, teledildon- 
ics, HR sex robots and VR sex games. A good overview of this is 
provided by Schermer and Ham.!!6 Galaitsi et al give an overview of 
the large number of dangers in this field. These include addiction, 
social isolation, non-consensual replication of real people, synthetic 
technology-enabled misogyny, racism and paedophilia. Galaitsi et al 
indicate that, so far, regulators seem to be shying away from setting 
clear legal frameworks, which they stress are necessary.''” Gieseke has 
indicated that current laws are often unable to adequately address the 
dangers arising from DF pornography.''® Meanwhile, Danaher has 
called for regulation over the use of child sex robots, for example, in 
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the treatment of paedophiles.!!? Although it is clear that many forms of 
non-consensual DF or virtual porn are simply prohibited under crimi- 
nal law, Sparrow and Karas indicate that the definition of rape may 
need to be reconsidered if someone’s synthetic likeliness is used for the 
purposes of teledildonics without their consent. ° 


6.4. DILEMMAS FOR THE REGULATOR 


Although it is clear that there are many tensions that regulators need 
to deal with in view of SR, there is no clear path forward. The biggest 
uncertainty that regulators need to account for is how AI and synthetic 
technologies will work in practice, if at all. Although there is reason to 
believe that this time it will be different, their success is far from a given. 
Moreover, even in those areas where AI is currently effective and evolving 
rapidly, there are signs that this evolution may soon come to a halt. For 
example, there may already be so much synthetic content available that 
Al systems trained on contemporary (post-2022) data sets are not learn- 
ing from humans or real life, but are in fact learning from themselves, 
thus creating a feedback loop, which may hamper progress or even bring 
it to a halt (model collapse). 

So, the first question is this: should the legislator take regulatory 
action? If so, this entails regulating partially blind, basing decisions on 
incomplete information.!*? This leads to the second question: if regula- 
tory action should be taken, when would be most opportune? 

On the one hand, it is clear that setting restrictions now might prevent 
considerable harm in the future. To draw an analogy, the general consen- 
sus is that if we had known how social media could negatively impact 
the health of children,!” increase polarisation and segregation and drive 
the post-truth society when it was first introduced, strict rules would 
have been set. The same applies to the internet in general, which was left 
unregulated for decades, providing a safe haven for political dissidents, 
whistleblowers and marginalised groups, but also for sexual predators, 
cyberbullies, hackers and exploitative business models. Regulatory 
control would potentially have steered the development of the internet 
and social media towards a safer, more inclusive space that is respectful 
of human rights. It may also be necessary to tackle technologies at their 
introduction, because once the genie is out of the bottle and the technol- 
ogies are widely adopted, it may be almost impossible to put it back in. 

On the other hand, at this point, it is unclear what potential effects 
and harms will result from synthetic technologies. It may make sense 
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to wait until these effects actually materialise so that it is clearer on 
which points regulatory action should be taken, rather than tackling 
hypothetical effects that may never materialise. Also, setting governmen- 
tal standards now might stifle innovation and dissuade other forms of 
norm-setting. 

This leads to the third question: how. One regulatory option is to 
introduce governmental regulation, either through laws, bylaws or offi- 
cial guidelines. However, there are other forms of regulation too, such as 
setting norms through self-regulation. Currently, the industry seems to be 
too fragmented and too much in competitive mode to set shared stand- 
ards and rules itself, but it is not unknown (in fact, quite the opposite) 
for industries to set up codes and self-regulatory bodies after a period 
of bloom. Once an industry has gone through its first peak of innova- 
tion and competitiveness, and once the market has stabilised and parties 
have an effectively demarcated position, the shared interest in being seen 
as reliable and trustworthy typically becomes more important than the 
detriment of having to abide by rules.!*4 Rather, industries from medi- 
eval guilds to present-day notaries and lawyers have set their own vetting 
procedures, with disciplinary sanctions for those that violate the internal 
codes and ethical standards. !*° Setting legal standards can undermine the 
potential for sectors to regulate themselves, and it puts a high burden on 
the government to regulate and oversee, which is costly, often less granu- 
lar and can be less effective. 

Another option would be to simply leave the regulatory process to the 
market. Consumers could favour the parties that respect their interests 
and offer ethical AI services, giving these parties a competitive advan- 
tage over others. One example is the field of privacy and data protection, 
where there are signs that companies and start-ups start building privacy- 
friendly business models and advertise being privacy-friendly. It is clear 
that this trend has been triggered in part by the adoption of strict regula- 
tions like the GDPR. Although it is true that the laws regulating AI have 
not yet gained momentum, similar to the prominence and coverage given 
to the GDPR received after its introduction, the situation may change 
once the AI Act is finally adopted. 

On the one hand, it may be argued that the position of citizens is 
too weak in comparison to the large businesses that operate AI systems 
and that they are often locked-in, unable to move to other providers. In 
addition, it can be said that the small number of American companies 
that are leading in the field of AI have unprecedented resources, power 
and knowledge. Not for nothing have there been many calls to break up 
tech firms.!* Still, on the contrary, many AI systems are not consumer 
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products as such, but are used by professional parties and governmental 
organisations, which are in a better position to make demands of these 
tech firms or change provider. In addition, AI seems less controlled by a 
small number of tech giants than was the case with previous technology, 
if only because China is a powerful driver of innovation in AI modelling 
as well and because AI models have been made available open source and 
have consequently been democratised. 

Lawrence Lessig famously distinguished between four ways to regu- 
late: through law, through self-regulation, through the market and through 
technology itself. As has been explained, technology is not neutral: there 
are norms embedded in it. There are choices over what data are collected 
and fed to the algorithm, choices in terms of how the algorithm and the 
decision trees are formulated, choices on how the technology is applied and 
where, in the interpretation of results and so forth. If anything, the litera- 
ture overview (see section 6.3) showed that AI is approached in two ways: 
it poses a danger, but it presents opportunities too. Social norms can be 
programmed into AI, such as to reduce societal segregation through posi- 
tive discrimination. Although it may be hard for the regulator to impose 
technological standards, when it does not, this would not mean that no 
standards are embedded in technology, but rather that those standards are 
determined by commercial enterprises with no democratic legitimacy and 
that those standards may not necessarily be conducive to individual auton- 
omy, societal equality and the rule of law. 


Regulation 


Figure 6.2 Four ways to regulate!”’ 
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The fourth and the fifth questions are who should be regulated and 
where; these questions are interrelated. 

The problem with synthetic technologies is that there are usually many 
parties involved in their development and application. These parties are also 
often based in different jurisdictions. Next to the complicated value chain 
in terms of developing and deploying synthetic technologies, as discussed 
in section 5.5.3, there is an intricate value chain in terms of data, as there 
are typically multiple parties involved in gathering data and categorising 
them, in data training, enriching data sets, doing baseline measurements, 
testing the effectiveness of programs and creating feedback systems. 8 The 
current regulatory regimes leave those complex value chains intact. The Al 
Act, for example, lays down obligations for distributors, providers, opera- 
tors, importers, users and even third parties, all of them having their own 
separate responsibilities in the value chain. Admittedly, the GDPR includes 
a relatively hierarchical structure between the controller and the processor, 
the controller being the party that determines the goal and means for data 
processing and is thus responsible for the whole process, including the deal- 
ings by the processor, who processes data on behalf of the data controller. 
Yet, when it comes to synthetic technologies, many parties involved may 
be considered a data controller. The GDPR suggests that in complex value 
chains with multiple parties, each having its own motives for and means 
of processing personal data, every party is considered a joint controller, 
meaning they share responsibility to ensure respect for the GDPR. There 
are several regulatory options, besides leaving the contemporary regula- 
tory approach intact: 


e An initial option would be to opt for a simplified approach, so to have 
one party legally responsible and liable for the whole value chain. 
This would have the additional advantage that the European regula- 
tor could lay the responsibility on a Europe-based entity. In turn, this 
would ensure better accountability, oversight and enforceability. 


e A second option is to nationalise parts of the AI sector. Currently, 
the AI market is generally in the hands of private sector organisa- 
tions. This means governmental regulation is by definition reactive. 
Although nationalisation is a radical step, it is not unprecedented 
from a historical perspective,'*? and could be considered owing to the 
enormous societal significance and impact of synthetic technologies. 


e <A third option could be to have the government fund or develop AI 
systems for public use ensuring, inter alia, that societal values are 
embedded in them by design. Although there is a myth that the public 
sector is not innovative and not as efficient as the private sector, espe- 
cially when it comes to technology, this belief is largely unfounded. 1*0 
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This option would also address the concern with the current status 
quo that there is an enormous knowledge gap between the public and 
the private sector, which means that the former is becoming increas- 
ingly dependent on the latter. 


A fourth option would be to keep the diversified approach intact, 
but to impose more, clearer and stricter rules on one or more of the 
parties involved. 


A fifth and final option would be to break up the larger tech firms that 
currently dominate the field of synthetic technologies, which would 
have the advantage of breaking up quasi-monopolies, but the disad- 
vantage that liability and responsibility may be even more difficult to 
assign. 


The sixth question is what the object of regulation should be. Under 
the current regulatory paradigm, there is a mix of approaches. Roughly 
speaking, four non-exhaustive types of regulatory objects can be distin- 
guished for the purposes of synthetic technologies, which partly overlap 


and do not always align smoothly: 


131 


The primary approach in the current regulatory framework is to 
focus on the effects of data processing and the use of technology. 
Harm, for example, may be addressed through civil and criminal law, 
such as when intellectual property rights are violated, when fraud is 
committed through identify theft, when an HR does physical harm 
to a patient or when AR devices block important information while 
navigating the road. To a large extent, the human rights regime is also 
effects-based insofar as when the effect of a technology is a privacy 
violation, interferes with the freedom of speech or is discriminatory, 
a citizen may invoke their right. 


A second approach is to focus on preventing harm and on the harm 
triggered by the possibility of an interference, or what is called the 
chilling effect. The European Court of Human Rights has recently 
placed greater emphasis on preventing chilling effects, procedural 
requirements and legislative checks and balances that are necessary 
in the light of potential future harm.'** This means that the chilling 
effect itself can be deemed an interference with a human right and that 
not having adequate safeguards in place against the arbitrary use of 
power, and not only the actual abuse of power, can lead to a violation 
of the European Convention on Human Rights. From a philosophi- 
cal perspective, this can be seen as a shift from an understanding of 
freedom as non-interference to an understanding of freedom as non- 
domination.” Outside the human rights framework, there is a focus 
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on preventing harm as well (eg, through impact assessments required 
by the GDPR) and there are several procedural safeguards, such 
as those provided in the AI Act. Although this approach is clearly 
emerging within the European legal framework, it is an approach that 
is still significantly less visible than the first approach noted above. 
Given the potential impact of synthetic technologies, this second 
approach may need to become more prominent. 


e A third approach is to regulate specific technologies and their use. 
There are some rules in the AI Act that take this approach, such as 
those laying down prohibitions for the use of AI systems for certain 
purposes and settings. There are also rules on the export of technolo- 
gies and components in the Dual Use Regulation. However, apart from 
these, the European legislator has thus far mostly chosen to adopt 
technology-neutral regulation.'*+ Technology-neutral regulation has 
several advantages: it becomes outdated less quickly than technology 
specific regulation and is more difficult to circumvent (eg, tech- 
specific regulation can be circumvented by redesigning technology to 
fall outside the scope of its legal definition). It also gives leeway to 
courts and supervisory authorities to interpret principles in the spirit 
of the law, when faced with a new, unforeseen technology. However, 
there are also disadvantages. Technology-neutral regulation tends to 
come at the cost of legal certainty. This may, in turn, lead to a chilling 
effect for citizens and benevolent organisations that want to abide by 
the legal principles, but do not know how to do so. In addition, tech- 
nology-neutral regulation always allows for discussion over the exact 
interpretation and application of the rules to a specific technology. 
This means that malevolent actors might use that ambiguity to their 
advantage. A final argument for adopting more technology specific 
rules is that the synthetic systems and their effects are very different. 
DF pornography is incomparable to a biased HR care robot perfor- 
mance. VR avatar theft is substantially different from the danger of 
consolidation of truth through chatbots and AR information devices 
that limit themselves to reproducing scientific and societal consensus. 


e A fourth approach is the regulation of data, which is increasingly 
important in the EU. Of course, the GDPR lays down standards for 
processing personal data, but there are various new instruments as 
well, such as the Regulation on the transfer of non-personal data, the 
proposed Data Act and the Data Governance Act. These instruments 
try to set standards for various types of data. The upcoming e-Privacy 
Regulation includes specific rules for processing metadata. Although 
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there is no unified approach to data regulation, one thing stands out: 
that under the classic distinction between stages of data operations 
in (1) gathering, (2) processing and (3) using data, almost all of the 
rules refer to the first stage: the gathering and storing of data (eg, 
legality, the processing ground, purpose specification, data minimisa- 
tion, technical and organisational security measures) or are linked to 
that moment (eg, the purpose limitation principle is connected to the 
purpose indicated at the moment of data collection, as is the storage 
limitation principle). It might be an option to lay down more rules 
on the processing and use of data, to the extent that these do not yet 
follow from the first and second approach as discussed above. 


This leads to the seventh and final question: why? This question is particu- 
larly difficult to answer in terms of synthetic technologies. This is because 
their practical effect is unknown, because the technologies may have complex 
multi-layered effects, because two opposite effects may materialise at the 
same time and because what counter-measures, if any, should be adopted is a 
matter of debate. Taking regulatory action can be especially difficult because 
choosing to protect one value may come at the cost of another. Pushing for 
more stability, reliability and repeatability of AI systems, for example, may 
come at the cost of diversification, serendipity and granularity. It may also be 
complex because it requires an understanding of technology and its effects 
on society as well as of human nature and the role of government. 

Take the transparency principle as an example. It could be interpreted 
as directed at citizens, as a part of their informational self-determina- 
tion, this is deemed to reduce the prevalent information imbalance and 
enhance personal autonomy. The question is what an individual can and 
should do with the information. Is it realistic to require a layperson to 
understand what the innumerable parties processing their data actually 
do with their information? Is it realistic to expect them to comprehend 
how synthetic technologies affect their life? Is it realistic to expect them 
to ascertain whether the activities of those parties are lawful and, if 
not, to take legal action against companies that have seemingly endless 
resources or a data provider based overseas? However, the transparency 
principle could also be understood as primarily allowing supervisory 
organisations to protect citizens’ interests by having all relevant informa- 
tion at their disposal. Under this approach, questions will emerge as well, 
such as, but not limited to: is a supervisory authority that acts against an 
organisation that processes personal data unlawfully enhancing citizens’ 
autonomy or would it amount to paternalism, such as when citizens are 
OK with data processing, even though it may not respect all the legal 
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requirements? In a third of many other possible interpretations, the 
transparency principle is seen as primarily directed at the public at large, 
to increase awareness, allow for an informed societal debate and enable 
civil right groups to commence legal proceedings. Each interpretation of 
the transparency principle requires different information to be provided 
to different actors through different means and in a different format. 

The additional complexity is that most Al-driven technologies are 
aimed at the core of human nature, which is essentially conflicted.! A 
person may, for example, rationally wish to read a book at night, drink a 
cup of tea and go to bed early, but there is something primitively desirable 
about drinking a glass of wine and binge-watching a Netflix series into 
the early hours. We often do not know what we actually want ourselves, 
or sometimes only years later; we often regret decisions we make, includ- 
ing decisions that we were convinced aligned with our authentic beliefs 
and ambitions at the time. We often need the help of others and technol- 
ogy to meet our own goals, such as having an app on our phones to push 
us to take a certain number of steps per day or asking our partner to 
keep us from drinking more than two glasses of alcohol at night. Human 
nature is truly complex and it is often impossible to say who a person 
‘really’ is or what a person’s authentic self is, even for the person them- 
selves. However, it is impossible to determine what enhances autonomy 
or what amounts to ethical AI before answering which of the various 
desires and conflicted selves, if any, reflects our true self. 136 

Does an HR sex robot enhance a person’s autonomy by giving them 
all the sex they desire or does it undermine their opportunity to develop 
a mature human relationship? Is DF technology that allows a person to 
continue talking to their deceased spouse good because it allows them to 
seek comfort in difficult times or bad because it keeps them away from 
what is best for them, what they (perhaps unknowingly) actually want: 
to confront the pain and learn to live with it? Is an AR device that blocks 
blood from a child’s sight laudable because it protects the child from 
harmful scenes that may lead to trauma, or does it undermine the child’s 
autonomy because they do not learn to deal with shocking or disgusting 
stimuli by themselves and thus become dependent on the AR service? 
Does VR technology that allows us to explore new identities feed into 
our primitive desires to embody an ideal avatar keep us away from work- 
ing on our true self? Or does it enhance a person’s autonomy as it allows 
them to explore deeply felt desires and new identities, which they may 
not be able to do in real life? In the end, most questions surrounding 
technology boil down to questions about human nature, meaning that 
there are often no correct answers. 
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Given that the answers to the many questions, dilemmas and intricacies 
that the regulator needs to address depend on personal, moral and politi- 
cal convictions, it is impossible to provide ready-made solutions here. 
However, it is possible to suggest points that need to be addressed and 
the order in which they need to be dealt with. So far, the regulator has 
always been two steps behind technological developments. It is typically 
only years after the introduction of a technology that legislators consider 
the need to inform themselves about its potential effects and start taking 
regulatory action; by then, it is often too late. There are signs that, with 
Al, things will be different, as regulators across the globe are organising 
Al information sessions, issuing standards for Generative AI and adopt- 
ing regulatory packages for synthetic technologies. This is important 
given the potential impact of the technology. This section provides a non- 
exhaustive list of points the regulator needs to consider and the order in 
which they should be considered. 

First, the regulator needs to decide whether the various relevant 
parties need more information. Obviously, citizens need to be informed 
about the potential dangers of synthetic technologies, how they can use 
Generative AI in their personal lives and what rights they have in rela- 
tion to the organisations that use these technologies. There also need 
to be awareness campaigns to inform organisations about the rules they 
need to comply with when developing, marketing and using synthetic 
technologies. This is because there is considerable ignorance surround- 
ing the existing and upcoming legal frameworks, especially with oversees 
organisations that develop AI for the European market. An additional 
problem is that the current legal framework, even the upcoming AI Act, 
mostly contains fairly general rules and principles. This creates legal 
uncertainty, both for citizens affected by synthetic technologies and for 
organisations developing or using them, which stifles innovation. The 
regulator should consequently invest in articulating bylaws, guidelines 
and standards, specifying per technology, application or sector how the 
rules should be interpreted and what they mean in a specific context. 

In particular, the regulator should strengthen its own information 
position. !%” In the EU, it is primarily civil servants who have attained a 
high level of expertise in relation to AI and synthetic technologies. At 
the national level, this expertise is often scattered throughout the civil 
service, if it is present at all. In particular, at both the European and 
national levels, most politicians and decision makers have insufficient 
knowledge to debate these technologies properly, much less regulate 
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them. This allows lobbyists and representatives of tech companies to 
have a significant influence on the legislative process; it also means that 
politicians easily get carried away with media trends, having the effect 
that they often target specific incidents rather than structural problems. 
An additional problem that the narrative around AI, as with many other 
technologies, is non-factual, but hovers around either extreme utopian or 
dystopian thinking,!** such as AI techno-solutionism!? and the fear that 
AI will eventually wipe out the human race.!#° To prevent this polarity 
from misinforming the regulatory debate, more detailed and practical 
knowledge on the workings of Al is required. 

A second point involves a strategic decision. Synthetic technologies 
and Generative AI are of great public interest, given their importance 
for national security issues, the economy and society at large. This 
means that importing technologies and data may create dependencies 
on foreign powers. Exporting technologies and data may reduce any 
potential competitive advantages. In order to attain European strategic 
autonomy, a strong technological portfolio is vital.!4' This is why the 
need for further export and import bans may need to be evaluated in 
light of long-term strategic considerations. 

An even bigger danger than the dependency on foreign powers may be 
dependency on tech firms. To close this gap, the regulator should consider 
taking the development of Generative AI under its wing by adopting the 
role of innovator, by setting up an independent foundation tasked and 
funded with this goal or by incentivising the market to develop technolo- 
gies that are respectful of the rule of law, with parties having to share the 
design of the technology and the knowledge gained through develop- 
ing it. Mazzucato, Schaake and others have suggested that the regulator 
should actively intervene to shape the AI market.!** 

A third point is that the regulator has largely shied away from intro- 
ducing technical requirements and bans on technologies, in an attempt 
to retain technology-neutral legislation. The regulator should consider 
adopting specific technological standards of three types. The first is tech- 
nological security standards. For some products, such as cars, there is 
an established system of security standards, tests and evaluations; this is 
mostly lacking for Generative AI and synthetic technologies. Given the 
vulnerability the backdoors and other technological weaknesses create, 
concrete technical security standards could help reduce vulnerabilities 
and prevent against hacking from criminals or foreign powers. Second, 
setting technological standards may also ensure respect for human rights 
by design.'*? These types of standards should also be considered in 
light of Generative AI, to embed concerns over privacy, data protection, 
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non-discrimination and other human rights into the technical designs 
of systems. Third, the need for imposing bans or moratoria should also 
be assessed. Although it is sometimes falsely suggested that digital tech- 
nology cannot be regulated, much less banned, given the cross-border 
nature and complex value chain involved, the GDPR has shown this to 
be untrue. As with a ban on non-authorised possession of firearms in 
force in most European countries, which is not absolutely enforced but 
significantly reduces the number of people with firearms, bans on tech- 
nologies are not absolutely effective, but they reduce their prominence 
significantly. 

Subsequently, the regulator should assess a fourth point — whether it 
wants to formally apply the precautionary principle to AI: 


The precautionary principle is an approach to risk management, where, if 
it is possible that a given policy or action might cause harm to the public or 
the environment and if there is still no scientific agreement on the issue, the 
policy or action in question should not be carried out.'** 


Article 191 of the Treaty on the Functioning of the European Union 
specifies that EU policy on the environment shall aim at a high level of 
protection and be based on the precautionary principle and on the prin- 
ciples that preventive action should be taken, that environmental damage 
should as a priority be rectified at source and that the polluter should 
pay.'4> Although it is clear that applying this principle mutatis mutandis 
to synthetic technologies could have a negative impact on innovation, !* 
at least in the short term, given the potential impact of Generative AI, it 
should be considered seriously. In the long term, clear and strict regula- 
tory standards may in fact boost innovation. 

Fifth, the current regulatory framework, such as the GDPR and the 
proposed AI Act, to the extent that it is focused on preventing harm, is 
concerned with the specific rather than the general on at least two points. 
First, it focuses primarily on the impact of systems and data processes on 
citizens, but less so on societal interests. Second, it takes account of the 
effects of one specific system or data process, not the combined effects 
thereof.'4” Both points may need to be reconsidered. Societal interests 
are affected through the combined effects of synthetic technologies. 4% 
A single application will not in and of itself amount to a post-truth society, 
undermine interpersonal trust or deepen societal inequality, but a pleth- 
ora of effects combined may do so. This may require a governmental 
organisation to be involved in the evaluating the effects of Generative AI. 
Assessing the permissibility of an individual system or specific applica- 
tion in light of its combined effects and societal implications in relation 
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to existing or proposed systems or applications requires an entity with an 
overview of the whole sector.!” 

The sixth point the regulator needs to account for is the effectiveness 
of Al systems. There is a persistent yet unfounded belief in new technolo- 
gies and their ability to solve problems or optimise processes. This means 
that millions are spent on projects, only to pull the plug a year or two 
later. Under the human rights framework, data-driven applications can 
only be introduced if there is reason to believe that they would increase 
effectiveness or efficiency in light of a legitimate aim. If a technology or 
system is not or is not very effective in relation to the intended goal, it 
will not be in conformity with the necessity, proportionality and subsidi- 
arity requirements.!°° These rules apply to public and private sector 
bodies alike when a human right is affected. Should the regulator want 
to allow organisations to use systems without established effectiveness, it 
should at a minimum consider imposing a sunset clause.'>! It may require 
organisations to first make baseline measurements to assess what the 
status quo is before introducing the synthetic technology, to subsequently 
determine benchmarks for the system and to finally assess whether those 
benchmarks are met. If the project does not deliver on its promise before 
the term in the sunset clause ends, it should be terminated. 15? 

Subsequently, the regulator needs to address a seventh point: whether 
additional legal obligations and requirements should be implemented. 
Many harmful activities are already prohibited or sanctioned through 
law, whether committed through traditional means or through synthetic 
technologies. However, on other points, additional rules may be needed, 
which have been discussed in detail. These include the questions of 
whether virtual rape should be criminalised, whether spreading DF news 
(particularly during election campaigns) should be prohibited, whether 
Al-driven entities should have fundamental rights of their own if they 
become sentient and whether, in light of the post-truth society, obliga- 
tions should be imposed on internet providers to monitor and block 
inauthentic content. It also needs to be assessed whether the procedural 
rules on the admissibility of evidence in court need to be revised and 
whether a digital forensic institute should be set up that assesses all mate- 
rial introduced in court by default. 

The eighth step is strengthening supervisory authorities. Mention 
was made of two potential new government bodies: an authority that 
assesses the admissibility of synthetic technologies and concrete applica- 
tions with an eye on combined effects, societal interests and long-term 
effects, and a digital forensic institute that assesses evidence submitted in 
court by default. If created, both would require a substantial budget and 
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resources given the potential volume of synthetic media and the role of 
Generative AI. It is equally clear that existing governmental bodies may 
need to be better equipped in light of the many challenges that arise in 
the synthetic society. These bodies include the public prosecution office, 
the Data Protection Authority and the various supervisory authorities 
that oversee the financial and other specific sectors, as well as those that 
enforce competition, consumer and telecommunications law. If AI is 
able to deliver on its promise of efficiency and effectiveness, part of the 
budget savings should be redirected to ensuring oversight on all relevant 
safeguards. 

Ninth, the regulator should invest in international cooperation in 
terms of enforcement and norm-setting. For organisations that develop 
and market synthetic technologies and Generative Al, which generally 
operate in multiple jurisdictions, the regulatory frameworks with which 
they need to comply should be as consistent as possible. The European 
regulator has a strategic choice to make. Does it want to pursue the 
traditional route of international norm-setting, through bodies such as 
the UN or the OECD? Or, alternatively, should it aim to become the 
world’s AI regulator through the Brussels effect?! In addition to strict 
governmental oversight, the regulator may need to seek cooperation with 
industry sectors.!** The GDPR encouraged sectors to adopt codes of 
conduct and set up sectoral supervisory bodies, but both because of the 
administrative and financial burden this puts on sectors, and because the 
advantages were considered to be too small, very few codes of conduct 
have been adopted and approved. To make this approach a success with 
respect to synthetic technologies, the regulator should consider either 
providing more substantial advantages to sectors that maintain oversight 
themselves or impose a duty on several sectors so that they do so. 

Tenth and finally, the regulator should assess the need for a bigger 
punitive stick, next to the carrot of investing in or subsidising organisa- 
tions that develop human centred synthetic technologies (see point two 
above). A persistent problem in the European legal context is that when 
citizens go to court (which may be costly and time-consuming) and win 
a case, they often only receive damages of hundreds or maybe one or 
two thousand euros, even when it concerns violations of bodily integrity. 
In terms of fairness and deterrent effect, the regulator should consider 
increasing damages in certain settings. Given the large societal interest at 
stake, account should also be made of the fines and penalties that regula- 
tory bodies can impose. The GDPR introduced the possibility to impose 
fines of €20 million or, in the case of an undertaking, up to four per 
cent of the total overall annual turnover of the preceding financial year, 
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which should also be considered by the regulator in light of Generative 


AI and synthetic technologies for contexts falling outside the scope of 
the GDPR. 


* Strengthen the information position of citizens, AI companies and the regulator in particular. 
Create more clarity about what the legal rules mean in light of synthetic technologies. 


* Consider creating public alternatives to privately owned and critical AI as well as export and 


import restrictions to maintain strategic autonomy. 
2. Strategic 


+ Assess the need for laying down technical requirements (security and human rights by design) 

Hace: and for bans or moratoria on synthetic technologies, either in full or in specific settings. 
moratoria and 
techoolegical 
requirements 


* Consider transposing the precautionary principle to generative AI and synthetic technologies. 
4, 
Precautionary 
principle 


e Ensure that societal interests and the cumulative effects of synthetic technologies are accounted 
for. 


+ Effectiveness needs to be proven before a synthetic technology is introduced; if there is 
uncertainty about its effectiveness, a sunset clause can be required, along with a baseline 
measurement, benchmarks and an evaluation of the effectiveness at the end of the project. 


_—  *Changes to the legal regime may need to be made, inter alia, criminalising virtual rape, 
amending the rules and procedures regarding the admissibility of evidence in court and 
7-Legislative imposing monitoring or filtering obligations on internet intermediaries. 


e New supervisory authorities may need to be set up. Giving considerably more resources, 


manpower and means to existing supervisory bodies should also be considered. 
8: Supervision 


e Cooperation is necessary at an international level. The European regulator needs to make a 
choice between setting standards through international bodies or aiming at the Brussels effect. 
Sectors may be encouraged or required to set up sectoral codes of conduct and disciplinary 
9. Cooperation bodies. 


* Both the capacity of supervisory authorities to lay down fines and penalties as well as 
the damages awarded to citizens for harm induced by synthetic technologies should be 


evaluated and should be so high that they are fair and have a deterrent effect. 


Figure 6.3 Points the regulator needs to deal with when regulating the 
synthetic society 
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6.6. CONCLUSION 


This book has given a full overview of synthetic technologies, their effects 
on society and the potential need for regulatory change. 

It has taken as its premise the most likely hypothesis, namely that 
Generative AI will be a modest success, which means that on several 
fronts, it will improve existing and facilitate new processes, but equally 
that there are many limitations and dangers that plague AI. Rules should 
consequently be set to incentivise human-centred AI and to counter the 
detrimental effects of Generative AI and the four technologies central to 
this book: HRs, DFs, AR and VR. Although good regulatory options 
are very few and far between and there is no carved path forward, if the 
regulator succeeds, the world as we know it might modestly change for 
the better. But alas, utopia will still be a pipe dream. 

However, there is a possibility that Generative AI will not be the 
success it is now predicted to be. Should synthetic technologies not 
deliver on their promises, like many other technologies before them, 
opportunities may be missed, but life may continue to be good. We could 
still read books, though without virtual characters popping up, we could 
go for drinks with friends at an actual bar with an actual bartender, we 
could take the train to Rome and see the marvellous buildings with our 
own eyes, while a local tour guide with a slight Italian accent shows us 
around, and, at the end of the day, when the mood is right, make love to 
our non-robotic partner. 

There is, on the other hand, a possibility that Generative AI and 
synthetic technologies turn out to be more effective than is currently 
predicted and that its dangers and pitfalls could be overcome in full, 
through technical, societal or regulatory changes. AI would then, among 
many other things, allow for better diagnostic choices, be more effective 
and efficient in the execution of tasks, make daily life easier and more 
fun, and relieve us of most if not all forms of arduous labour. What 
would we do with all of our spare time? Who knows: perhaps we would 
take the time to read a book and have a drink with friends, go to Rome to 
walk down the Via Appia Antica, try a Tonnarelli Cacio e Pepe at a local 
restaurant, and late at night, have a romantic evening with our partner 
watching the sun set over the Lago Albano. 
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